locking¶

Module to manage distributed locks.

exception spicerack.locking.EtcdLockUnavailableError[source]¶

Bases: SpicerackCheckError

Exception raised when unable to acquire the etcd lock, the operation should be retried.

exception spicerack.locking.InvalidLockError[source]¶

Bases: LockError

Exception raised when the parameters for a lock are invalid.

exception spicerack.locking.LockError[source]¶

Bases: SpicerackError

Generic exception for errors of this module.

exception spicerack.locking.LockExistingError[source]¶

Bases: LockError

Exception raised if a lock with the same ID already exists for the given key.

exception spicerack.locking.LockUnavailableError[source]¶

Bases: SpicerackCheckError

Exception raised when unable to acquire the Spicerack lock, the operation should be retried.

exception spicerack.locking.LockUnreadableError[source]¶

Bases: LockError

Exception raised when unable to properly parse an existing lock from the backend.

exception spicerack.locking.LockUnwritableError[source]¶

Bases: LockError

Exception raised when unable to properly serialize the lock in order to save it in the backend.

class spicerack.locking.ConcurrentLock(concurrency: int, owner: str, ttl: int, uuid: str = <factory>, created: datetime.datetime = <factory>) → None[source]¶

Bases: object

A single concurrent lock object.

Parameters:

concurrency (int) -- how many concurrent locks can be acquired for the same key. If set to zero it means that there is no concurrency limit and an infinite number of clients can run concurrently, but they are tracked.
ttl (int) -- the time to live in seconds of the lock. If expired it will be automatically discarded.
owner (str) -- a way to identify the owner of the lock, usually in the form {user}@{hostname} [{pid}].
uuid (str, default: <factory>) -- the unique identifier of the concurrent lock.
created (datetime.datetime, default: <factory>) -- when the lock has been created, used for checking if the TTL has expired.

classmethod from_dict(lock_id: str, lock_obj: dict[str, str | int]) → spicerack.locking.ConcurrentLock[source]¶

Get a ConcurrentLock instance from a dict, suitable to be used to JSON deserialize.

Parameters:

lock_id (str) -- the concurrent lock unique identifier.
lock_obj (dict[str, typing.Union[str, int]]) -- the lock dictionary object.

Return type:

spicerack.locking.ConcurrentLock

Returns:

a concurrent lock instance.

Raises:

spicerack.locking.InvalidLockError -- if unable to parse the created datetime.

to_dict() → dict[str, str | int][source]¶

Returns the dict representation of the object suitable for JSON serialization.

Intentionally not using dataclasses.asdict() to skip self.uuid and to convert the created datetime to string.

Return type:: dict[str, typing.Union[str, int]]
Returns:: the object as dict with the created datetime converted to string.

update_created() → None[source]¶

Update the created time to now.

Return type:: None

class spicerack.locking.KeyLocks(key: str, locks: dict[str, spicerack.locking.ConcurrentLock] = <factory>) → None[source]¶

Bases: object

Manage a lock object for a given key with concurrent locks.

Parameters:

key (str) -- the lock full key path to be used in the backend.
locks (dict[str, spicerack.locking.ConcurrentLock], default: <factory>) -- the concurrent locks for the given key.

add(lock: spicerack.locking.ConcurrentLock) → None[source]¶

Add the concurrent lock to the lock object if all criteria are met.

Parameters:

lock (spicerack.locking.ConcurrentLock) -- the concurrent lock to add.

Raises:

spicerack.locking.LockUnavailableError -- when the max concurrency has been reached and is not possible to acquire a lock.
spicerack.locking.LockExistingError -- if a lock with the same ID already exists.

Return type:

None

classmethod from_json(key: str, json_str: str) → spicerack.locking.KeyLocks[source]¶

Create an instance of this class from a JSON string.

Parameters:

key (str) -- the lock full key path to be used in the backend.
json_str (str) -- the JSON serialization of the lock object.

Return type:

spicerack.locking.KeyLocks

Returns:

the lock instance.

Raises:

spicerack.locking.LockUnreadableError -- if unable to decode the JSON lock.

remove(lock_id: str) → spicerack.locking.ConcurrentLock | None[source]¶

Remove the concurrent lock identified by the lock ID.

Parameters:: lock_id (str) -- the ID identifying the lock.
Return type:: typing.Optional[spicerack.locking.ConcurrentLock]
Returns:: the removed concurrent lock if the lock was found and removed from the instance, it means that the backend needs to be updated. None if the concurrent lock was not found and there is no need to update the backend.

to_json() → str[source]¶

Return the JSON serialization of the current instance.

Return type:: str
Returns:: a string with a JSON-serialized object.
Raises:: spicerack.locking.LockUnwritableError -- if unable to convert the curent instance to JSON.

class spicerack.locking.Lock(*, config: dict, prefix: str, owner: str, dry_run: bool = True) → None[source]¶

Bases: object

Manage a Spicerack lock.

Example object created in the backend (etcd):

'/spicerack/locks/cookbooks/sre.foo.bar' => {
    '4e2677c7-541a-4f9e-afbb-cfdb69c440a1': {
        'created': '2023-07-16 16:46:52.161053',
        'owner': 'user@host [12345]',
        'concurrency': 5,
        'ttl': 120,
    }
}

Initialize the instance.

Parameters:

config (dict) -- the etcd configuration dictionary to be passed to the etcd client.
prefix (str) -- the name of the directory to use to prefix the lock. Must be one of spicerack.locking.ALLOWED_PREFIXES.
owner (str) -- a way to identify the owner of the lock, usually in the form {user}@{hostname} [{pid}].
dry_run (bool, default: True) -- whether this is a DRY-RUN.

Raises:

spicerack.locking.LockError -- if the provided prefix is not one of the allowed ones.

acquire(name: str, *, concurrency: int, ttl: int) → str[source]¶

Acquire a lock for the given name and parameters and return the lock identifier.

Examples:

# Get a lock for a given name allowing to have other 2 concurrent runs in parallel with a TTL of 30 minutes
lock_id = lock.acquire("some.lock.identifier", concurrency=3, ttl=1800)
# Do something
lock.release("some.lock.identifier", lock_id)

# Acquire an exclusive lock for the given name with a TTL of 1 hour
lock_id = lock.acquire("some.lock.identifier", concurrency=1, ttl=1800)
# Do something
lock.release("some.lock.identifier", lock_id)

Parameters:

name (str) -- the lock name, cannot contain / as that's a directory separator in the data structure.
concurrency (int) -- how many concurrent clients can hold the same lock. If set to zero it means that there is no concurrency limit and an infinite number of clients can run concurrently, but they are tracked.
ttl (int) -- the amount of seconds this lock is valid for. All locks that have passed their TTL are considered expired and will be automatically removed.

Return type:

str

Returns:

the lock unique identifier.

acquired(name: str, *, concurrency: int, ttl: int) → Iterator[None][source]¶

Context manager to perform actions while holding a lock for the given name and parameters.

Examples:

# Get a lock for a given name allowing to have other 2 concurrent runs in parallel with a TTL of 30 minutes
with lock.acquired("some.lock.identifier", concurrency=3, ttl=1800):
    # Do something

# Acquire an exclusive lock for the given name with a TTL of 1 hour
with lock.acquired("some.lock.identifier", concurrency=1, ttl=1800):
    # Do something

Parameters:

name (str) -- the lock name, cannot contain / as that's a directory separator in the data structure.
concurrency (int) -- how many concurrent clients can hold the same lock. If set to zero it means that there is no concurrency limit and an infinite number of clients can run concurrently, but they are tracked.
ttl (int) -- the amount of seconds this lock is valid for. All locks that have passed their TTL are considered expired and will be automatically removed.

Yields:

Nothing, it just gives back control to the client's code while holding the lock.

Return type:

typing.Iterator[None]

get(name: str) → spicerack.locking.KeyLocks[source]¶

Get the existing locks for the given name. If missing returns a new object with no locks.

Parameters:: name (str) -- the lock name, cannot contain / as that's a directory separator in the data structure.
Raises:: spicerack.locking.LockError -- for any etcd errors beside the key not found one.
Return type:: spicerack.locking.KeyLocks
Returns:: the existing locks for the given name.

release(name: str, lock_id: str) → None[source]¶

Release the lock identified by the lock ID, best effort. The lock will expire anyway.

See the documentation for spicerack.locking.Lock.acquire() for usage examples.

Parameters:

name (str) -- the lock name, cannot contain / as that's a directory separator in the data structure.
lock_id (str) -- the ID identifying the lock.

Return type:

None

class spicerack.locking.NoLock(*_args: Any, **_kwargs: Any) → None[source]¶

Bases: object

A noop locking class that does nothing.

Has the same APIs of the spicerack.locking.Lock, to be used when locking support is disabled.

Initialize the instance.

Parameters:

*_args (typing.Any) -- accept any positional arguments that spicerack.locking.Lock.__init__() accepts.
**_kwargs (typing.Any) -- accept any keyword argument that spicerack.locking.Lock.__init__() accepts.

acquire(*_args: Any, **_kwags: Any) → str[source]¶

Dummy method that just returns an empty string.

Parameters:

*_args (typing.Any) -- accept any positional arguments that spicerack.locking.Lock.acquire() accepts.
**_kwargs -- accept any keyword argument that spicerack.locking.Lock.acquire() accepts.

Return type:

str

Returns:

an empty string.

acquired(*_args: Any, **_kwargs: Any) → Iterator[None][source]¶

Context manager that just yields.

Parameters:

*_args (typing.Any) -- accept any positional arguments that spicerack.locking.Lock.acquired() accepts.
**_kwargs (typing.Any) -- accept any keyword argument that spicerack.locking.Lock.acquired() accepts.

Yields:

Nothing, it just gives back control to the client's code.

Return type:

typing.Iterator[None]

get(name: str, *_args: Any, **_kwags: Any) → spicerack.locking.KeyLocks[source]¶

Dummy method that just returns an empty KeyLocks.

Parameters:

name (str) -- the lock name, cannot contain / as that's a directory separator in the data structure.
*_args (typing.Any) -- accept any positional arguments that spicerack.locking.Lock.acquire() accepts.
**_kwargs -- accept any keyword argument that spicerack.locking.Lock.acquire() accepts.

Return type:

spicerack.locking.KeyLocks

Returns:

an empty instance.

release(*_args: Any, **_kwags: Any) → None[source]¶

Dummy method that does nothing.

Parameters:

*_args (typing.Any) -- accept any positional arguments that spicerack.locking.Lock.release() accepts.
**_kwargs -- accept any keyword argument that spicerack.locking.Lock.release() accepts.

Return type:

None

spicerack.locking.get_lock_instance(*, config_file: pathlib.Path | None, prefix: str, owner: str, dry_run: bool = True) → spicerack.locking.Lock | spicerack.locking.NoLock[source]¶

Get a lock instance based on the configuration file and prefix.

Parameters:

config_file (typing.Optional[pathlib.Path]) -- the path to the configuration file for the locking backend or None to disable the locking support and return a spicerack.locking.NoLock instance. When the configuration file is present a spicerack.locking.Lock instance is returned instead. The configuration is also automatically merged with the ~/.etcdrc config file of the running user, if present.
prefix (str) -- the name of the directory to use to prefix the lock. Must be one of spicerack.locking.ALLOWED_PREFIXES.
owner (str) -- a way to identify the owner of the lock, usually in the form {user}@{hostname} [{pid}].
dry_run (bool, default: True) -- whether this is a DRY-RUN.

Raises:

spicerack.locking.LockError -- if the provided prefix is not one of the allowed ones.

Return type:

typing.Union[spicerack.locking.Lock, spicerack.locking.NoLock]

Returns:

The locking instance or a dummy instance that has the same API of the locking one but does nothing if the config_file.

spicerack.locking.ALLOWED_PREFIXES: tuple[str, ...] = ('cookbooks', 'custom', 'modules')¶: The allowed values for the prefix parameter to be used as path prefix for the locks keys.

spicerack.locking.COOKBOOKS_CUSTOM_PREFIX: str = 'custom'¶: The path prefix to use for the key of locks acquired from inside a cookbook.

spicerack.locking.COOKBOOKS_PREFIX: str = 'cookbooks'¶: The path prefix to use for the key of locks acquired by spicerack for each cookbook execution.

spicerack.locking.ETCD_WRITER_LOCK_KEY: str = 'etcd'¶: The path prefix of the short-term keys to acquire an exclusive lock to write to etcd.

spicerack.locking.KEYS_BASE_PATH: str = '/spicerack/locks'¶: The base path for the lock keys.

spicerack.locking.SPICERACK_PREFIX: str = 'modules'¶: The path prefix to use for the key of locks acquired from inside spicerack modules.