decorators

Decorators module.

class wmflib.decorators.RetryParams(tries: int, delay: timedelta, backoff_mode: str, exceptions: Tuple[Type[Exception], ...], failure_message: str)[source]

Bases: object

Retry decorator parameters class.

This class represents the parameters that the wmflib.decorators.retry() decorator accepts and allow to validate them.

It is also used for the dynamic_params_callbacks parameter of the same retry decorator to allow the callback to safely modify the decorator’s parameters at runtime.

validate() → None[source]

Validate the consistency of the current values of the instance properties.

Raises:: wmflib.exceptions.WmflibError – if any field has an invalid value.

wmflib.decorators.retry(func: ~typing.Callable, *, tries: int = 3, delay: ~datetime.timedelta = datetime.timedelta(seconds=3), backoff_mode: str = 'exponential', exceptions: ~typing.Tuple[~typing.Type[Exception], ...] = (<class 'wmflib.exceptions.WmflibError'>,), failure_message: str | None = None, dynamic_params_callbacks: ~typing.Tuple[~typing.Callable[[~wmflib.decorators.RetryParams, ~typing.Callable, ~typing.Tuple, ~typing.Dict[str, ~typing.Any]], None], ...] = ()) → Callable[source]

Decorator to retry a function or method if it raises certain exceptions with customizable backoff.

Note

The decorated function or method must be idempotent to avoid unwanted side effects. It can be called with or without arguments, in the latter case all the default values will be used.

Examples

Define a function that polls for the existence of a file, retrying with the default parameters:

>>> from pathlib import Path
>>> from wmflib.decorators import retry
>>> from wmflib.exceptions import WmflibError
>>> @retry
... def poll_file(path: Path):
...     if not path.exists():
...         raise WmflibError(f'File {path} not found')
...
>>> poll_file(Path('/tmp'))
>>> poll_file(Path('/tmp/nonexistent'))
[1/3, retrying in 3.00s] Attempt to run '__main__.poll_file' raised: File /tmp/nonexistent not found
[2/3, retrying in 9.00s] Attempt to run '__main__.poll_file' raised: File /tmp/nonexistent not found
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 160, in wrapper
    return func(*args, **kwargs)
  File "<stdin>", line 4, in poll_file
wmflib.exceptions.WmflibError: File /tmp/nonexistent not found
>>>

Same example, but customizing the decorator parameters:

>>> from datetime import timedelta
>>> from pathlib import Path
>>> from wmflib.decorators import retry
>>> @retry(tries=5, delay=timedelta(seconds=30), backoff_mode='constant', failure_message='File not found',
...        exceptions=(RuntimeError,))
... def poll_file(path: Path):
...     if not path.exists():
...         raise RuntimeError(path)
...
[1/5, retrying in 30.00s] File not found: /tmp/nonexistent
[2/5, retrying in 30.00s] File not found: /tmp/nonexistent
[3/5, retrying in 30.00s] File not found: /tmp/nonexistent
[4/5, retrying in 30.00s] File not found: /tmp/nonexistent
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/wmflib/decorators.py", line 160, in wrapper
    return func(*args, **kwargs)
  File "<stdin>", line 5, in poll_file
RuntimeError: /tmp/nonexistent

Parameters:

func (function, method) – the decorated function.
tries (int, optional) – the number of times to try calling the decorated function or method before giving up. Must be a positive integer.
delay (datetime.timedelta, optional) – the initial delay in seconds for the first retry, used also as the base for the backoff algorithm.

backoff_mode (str, optional) –

the backoff mode to use for the delay, available values are shown below. All the examples are made with tries=5 and delay=3:

constant

Nth delay   = delay
Total delay = delay * tries
Example:      3s, 3s,  3s,  3s,   3s => 15s max possible delay

linear

Nth delay   = (delay * N) with N in [1, tries]
Total delay = 0.5 * tries * (delay + (delay * tries))
Example:      3s, 6s,  9s, 12s,  15s => 45s max possible delay

power

Nth delay   = (delay * 2^N) with N in [0, tries - 1]
Total delay = delay * (2**tries - 1)
Example:      3s, 6s, 12s, 24s,  48s => 93s max possible delay

exponential

Nth delay   = (delay^N) with N in [1, tries], delay must be > 1
Total delay = (delay * (delay**tries - 1)) / (delay - 1)
Example:      3s, 9s, 27s, 81s, 243s => 363s max possible delay

exceptions (type, tuple, optional) – the decorated function call will be retried if it fails until it succeeds or tries attempts are reached. A retryable failure is defined as raising any of the exceptions listed.
failure_message (str, optional) – the message to log each time there’s a retryable failure. Retry information and exception message are also included. Default: “Attempt to run ‘<fully qualified function>’ raised”

dynamic_params_callbacks (tuple) –

a tuple of callbacks that will be called at runtime to allow to modify the decorator’s parameters. Each callable must adhere to the following interface:

def adjust_some_parameter(retry_params: RetryParams, func: Callable, args: Tuple, kwargs: Dict) -> None
    # Modify the retry_params parameter possibly using the decorated function object or its parameters
    # that are passed as tuple for the positional arguments and a dictionary for the keyword arguments

This is a practical example that defines a callback that doubles the delay parameter of the @retry decorator if the decorated function/method has a ‘slow’ keyword argument that is to True:

def double_delay(retry_params, func, args, kwargs):
    if kwargs.get('slow', False):
        retry_params.delay = retry_params.delay * 2

@retry(delay=timedelta(seconds=10), dynamic_params_callbacks=(double_delay,))
def do_something(slow=False):
    # This method will be retried using 10 seconds as delay parameter in the @retry decorator, but
    # if the 'slow' parameter is set to True it will use a delay of 20 seconds instead.
    # Do something here.

While the callbacks will have access to the parameters passed to the decorated function, they should be treated as read-only variables.

Returns:

the decorated function.

Return type:

function