Python Guidelines: Implementation

Configuration

✅ DO honor the following environment variables for global configuration settings:

Parameter validation

The service client will have several methods that send requests to the service. Service parameters are directly passed across the wire to an Azure service. Client parameters aren’t passed directly to the service, but used within the client library to fulfill the request. Parameters that are used to construct a URI, or a file to be uploaded are examples of client parameters.

✅ DO validate client parameters. Validation is especially important for parameters used to build up the URL since a malformed URL means that the client library will end up calling an incorrect endpoint.

# No:
def get_thing(name: "str") -> "str":
    url = f'https://<host>/things/{name}'
    return requests.get(url).json()

try:
    thing = get_thing('') # Ooops - we will end up calling '/things/' which usually lists 'things'. We wanted a specific 'thing'.
except ValueError:
    print('We called with some invalid parameters. We should fix that.')

# Yes:
def get_thing(name: "str") -> "str":
    if not name:
        raise ValueError('name must be a non-empty string')
    url = f'https://<host>/things/{name}'
    return requests.get(url).json()

try:
    thing = get_thing('')
except ValueError:
    print('We called with some invalid parameters. We should fix that.')

⛔️ DO NOT validate service parameters. Don’t do null checks, empty string checks, or other common validating conditions on service parameters. Let the service validate all request parameters.

✅ DO validate the developer experience when the service parameters are invalid to ensure appropriate error messages are generated by the service. Work with the service team if the developer experience is compromised because of service-side error messages.

Network operations

Since the client library generally wraps one or more HTTP requests, it’s important to support standard network capabilities. Although not widely understood, asynchronous programming techniques are essential in developing resilient web services. Many developers prefer synchronous method calls for their easy semantics when learning how to use a technology. The HTTP pipeline is a component in the azure-core library that assists in providing connectivity to HTTP-based Azure services.

✅ DO use the HTTP pipeline to send requests to service REST endpoints.

☑️ YOU SHOULD include the following policies in the HTTP pipeline:

• Telemetry (azure.core.pipeline.policies.UserAgentPolicy)
• Unique Request ID
• Retry (azure.core.pipeline.policies.RetryPolicy and azure.core.pipeline.policies.AsyncRetryPolicy)
• Credentials
• Response downloader
• Distributed tracing
• Logging (azure.core.pipeline.policies.NetworkTraceLoggingPolicy)

Dependencies

✅ DO only pick external dependencies from the following list of well known packages for shared functionality:

⛔️ DO NOT use external dependencies outside the list of well known dependencies. To get a new dependency added, contact the Architecture Board.

⛔️ DO NOT vendor dependencies unless approved by the Architecture Board.

When you vendor a dependency in Python, you include the source from another package as if it was part of your package.

⛔️ DO NOT pin a specific version of a dependency unless that is the only way to work around a bug in said dependencies versioning scheme.

Applications are expected to pin exact dependencies. Libraries aren’t. A library should use a compatible release identifier for the dependency.

Service-specific common library code

There are occasions when common code needs to be shared between several client libraries. For example, a set of cooperating client libraries may wish to share a set of exceptions or models.

✅ DO gain Architecture Board approval prior to implementing a common library.

✅ DO minimize the code within a common library. Code within the common library is available to the consumer of the client library and shared by multiple client libraries within the same namespace.

✅ DO store the common library in the same namespace as the associated client libraries.

A common library will only be approved if:

• The consumer of the non-shared library will consume the objects within the common library directly, AND
• The information will be shared between multiple client libraries.

Let’s take two examples:

1. Implementing two Cognitive Services client libraries, we find a model is required that is produced by one Cognitive Services client library and consumed by another Coginitive Services client library, or the same model is produced by two client libraries. The consumer is required to do the passing of the model in their code, or may need to compare the model produced by one client library vs. that produced by another client library. This is a good candidate for choosing a common library.

2. Two Cognitive Services client libraries throw an ObjectNotFound exception to indicate that an object was not detected in an image. The user might trap the exception, but otherwise will not operate on the exception. There is no linkage between the ObjectNotFound exception in each client library. This is not a good candidate for creation of a common library (although you may wish to place this exception in a common library if one exists for the namespace already). Instead, produce two different exceptions - one in each client library.

Error handling

✅ DO raise an exception if a method fails to perform its intended functionality. Don’t return None or a boolean to indicate errors.

# Yes
try:
    resource = client.create_resource(name)
except azure.core.errors.ResourceExistsException:
    print('Failed - we need to fix this!')

# No
resource = client.create_resource(name):
if not resource:
    print('Failed - we need to fix this!')

⛔️ DO NOT throw an exception for “normal responses”.

Consider an exists method. The method must distinguish between the service returned a client error 404/NotFound and a failure to even make a request:

# Yes
try:
    exists = client.resource_exists(name):
    if not exists:
        print("The resource doesn't exist...")
except azure.core.errors.ServiceRequestError:
    print("We don't know if the resource exists - so it was appropriate to throw an exception!")

# No
try:
    client.resource_exists(name)
except azure.core.errors.ResourceNotFoundException:
    print("The resource doesn't exist... but that shouldn't be an exceptional case for an 'exists' method")

⚠️ YOU SHOULD NOT create a new exception type unless the developer can remediate the error by doing something different. Specialized exception types should be based on existing exception types present in the azure-core package.

✅ DO produce an error when an HTTP request fails with an unsuccessful HTTP status code (as defined by the service).

✅ DO include the HTTP response (status code and headers) and originating request (URL, query parameters, and headers) in the exception.

For higher-level methods that use multiple HTTP requests, either the last exception or an aggregate exception of all failures should be produced.

✅ DO include any service-specific error information in the exception. Service-specific error information must be available in service-specific properties or fields.

✅ DO document the errors that are produced by each method. Don’t document commonly thrown errors that wouldn’t normally be documented in Python.

⛔️ DO NOT create new exception types when a built-in exception type will suffice.

✅ DO allow exception chaining to include the original source of the error.

# Yes:
try:
    # do something
except:
    raise MyOwnErrorWithNoContext()

# No:
success = True
try:
    # do something
except:
    success = False
if not success:
    raise MyOwnErrorWithNoContext()

# No:
success = True
try:
    # do something
except:
    raise MyOwnErrorWithNoContext() from None

Logging

✅ DO use Pythons standard logging module.

✅ DO provide a named logger for your library.

The logger for your package must use the name of the module. The library may provide additional child loggers. If child loggers are provided, document them.

For example:

• Package name: azure-someservice
• Module name: azure.someservice
• Logger name: azure.someservice
• Child logger: azure.someservice.achild

These naming rules allow the consumer to enable logging for all Azure libraries, a specific client library, or a subset of a client library.

✅ DO use the ERROR logging level for failures where it’s unlikely the application will recover (for example, out of memory).

✅ DO use the WARNING logging level when a function fails to perform its intended task. The function should also raise an exception.

Don’t include occurrences of self-healing events (for example, when a request will be automatically retried).

✅ DO use the INFO logging level when a function operates normally.

✅ DO use the DEBUG logging level for detailed trouble shooting scenarios.

The DEBUG logging level is intended for developers or system administrators to diagnose specific failures.

⛔️ DO NOT send sensitive information in log levels other than DEBUG. For example, redact or remove account keys when logging headers.

✅ DO log the request line, response line, and headers for an outgoing request as an INFO message.

✅ DO log an INFO message, if a service call is canceled.

✅ DO log exceptions thrown as a WARNING level message. If the log level set to DEBUG, append stack trace information to the message.

You can determine the logging level for a given logger by calling logging.Logger.isEnabledFor.

Distributed tracing

DRAFT section

✅ DO create a new trace span for each library method invocation. The easiest way to do so is by adding the distributed tracing decorator from azure.core.tracing.

✅ DO use <package name>/<method name> as the name of the span.

✅ DO create a new span for each outgoing network call. If using the HTTP pipeline, the new span is created for you.

✅ DO propagate tracing context on each outgoing service request.

Azure Core

The azure-core package provides common functionality for client libraries. Documentation and usage examples can be found in the azure/azure-sdk-for-python repository.

HTTP pipeline

The HTTP pipeline is an HTTP transport that is wrapped by multiple policies. Each policy is a control point that can modify either the request or response. A default set of policies is provided to standardize how client libraries interact with Azure services.

For more information on the Python implementation of the pipeline, see the documentation.

Custom Policies

Some services may require custom policies to be implemented. For example, custom policies may implement fall back to secondary endpoints during retry, request signing, or other specialized authentication techniques.

☑️ YOU SHOULD use the policy implementations in azure-core whenever possible.

✅ DO document any custom policies in your package. The documentation should make it clear how a user of your library is supposed to use the policy.

✅ DO add the policies to the azure.<package name>.pipeline.policies namespace.

Protocols

LROPoller

T = TypeVar("T")
class LROPoller(Protocol):

    def result(self, timeout=None) -> T:
        """ Retreive the final result of the long running operation.

        :param timeout: How long to wait for operation to complete (in seconds). If not specified, there is no timeout.
        :raises TimeoutException: If the operation has not completed before it timed out.
        """
        ...

    def wait(self, timeout=None) -> None:
        """ Wait for the operation to complete.

        :param timeout: How long to wait for operation to complete (in seconds). If not specified, there is no timeout.
        """

    def done(self) -> boolean:
        """ Check if long running operation has completed.
        """

    def add_done_callback(self, func) -> None:
        """ Register callback to be invoked when operation completes.

        :param func: Callable that will be called with the eventual result ('T') of the operation.
        """
        ...

azure.core.LROPoller implements the LROPoller protocol.

Paged

T = TypeVar("T")
class ByPagePaged(Protocol, Iterable[Iterable[T]]):
    continuation_token: "str"

class Paged(Protocol, Iterable[T]):
    continuation_token: "str"

    def by_page(self) -> ByPagePaged[T] ...

azure.core.Paged implements the Paged protocol.

DiagnosticsResponseHook

class ResponseHook(Protocol):

    __call__(self, headers, deserialized_response): -> None ...

Versioning

✅ DO use semantic versioning for your package.

✅ DO use the bN pre-release segment for beta releases.

Don’t use pre-release segments other than the ones defined in PEP440 (aN, bN, rcN). Build tools, publication tools, and index servers may not sort the versions correctly.

✅ DO change the version number if anything changes in the library.

✅ DO increment the patch version if only bug fixes are added to the package.

✅ DO increment the minor version if any new functionality is added to the package.

✅ DO increment the minor version if the default REST API version is changed, even if there’s no public API change to the library.

⛔️ DO NOT increment the major version for a new REST API version unless it requires breaking API changes in the python library itself.

✅ DO increment the major version if there are breaking changes in the package. Breaking changes require prior approval from the Architecture Board.

✅ DO select a version number greater than the highest version number of any other released Track 1 package for the service in any other scope or language.

The bar to make a breaking change is extremely high for GA client libraries. We may create a new package with a different name to avoid diamond dependency issues.

REST API method versioning

✅ DO use the latest service protocol version when making requests.

✅ DO allow a client application to specify an earlier version of the service protocol.

Packaging

✅ DO name your package after the namespace of your main client class.

✅ DO use all lowercase in your package name with a dash (-) as a separator.

⛔️ DO NOT use underscore (_) or period (.) in your package name. If your namespace includes underscores, replace them with dash (-) in the distribution package name.

✅ DO follow the specific package guidance from the azure-sdk-packaging wiki

✅ DO follow the namespace package recommendations for Python 3.x for packages that only need to target 3.x.

✅ DO depend on azure-nspkg for Python 2.x.

✅ DO include __init__.py for the namespace(s) in sdists

Binary extensions

✅ DO be approved by the Architecture Board.

✅ DO support Windows, Linux (manylinux - see PEP513, PEP571), and MacOS. Support the earliest possible manylinux to maximize your reach.

✅ DO support both x86 and x64 architectures.

✅ DO support unicode and ASCII versions of CPython 2.7.

Testing

✅ DO use pytest as the test framework.

☑️ YOU SHOULD use pytest-asyncio for testing of async code.

✅ DO make your scenario tests runnable against live services. Strongly consider using the Python Azure-DevTools package for scenario tests.

✅ DO provide recordings to allow running tests offline/without an Azure subscription

✅ DO support simultaneous test runs in the same subscription.

✅ DO make each test case independent of other tests.

Recommended Tools

✅ DO use pylint for your code. Use the pylintrc file in the root of the repository.

✅ DO use flake8-docstrings to verify doc comments.

✅ DO use Black for formatting your code.

☑️ YOU SHOULD use MyPy to statically check the public surface area of your library.

You don’t need to check non-shipping code such as tests.