`tools` — Miscellaneous Helper Functions#

Miscellaneous helper functions (not wiki-dependent).

Deprecated since version 7.6: The CombinedError, DequeGenerator, EmptyDefault, EMPTY_DEFAULT and SizedKeyCollection objects. Import them from tools.collections instead.

Deprecated since version 7.6: The itergroup function. Use backports.batched() or itertools.batched instead.

Deprecated since version 7.6: The filter_unique, intersect_generators, islice_with_ellipsis and roundrobin_generators functions. Import them from tools.itertools instead.

Deprecated since version 7.7: The RLock, ThreadedGenerator and ThreadList classes. Import them from tools.threading instead.

class tools.ComparableMixin[source]#

Bases: ABC

Mixin class to allow comparing to other objects which are comparable.

Added in version 3.0.

class tools.MediaWikiVersion(version_str)[source]#

Bases: object

Version object to allow comparing ‘wmf’ versions with normal ones.

The version mainly consist of digits separated by periods. After that is a suffix which may only be ‘wmf<number>’, ‘alpha’, ‘beta<number>’ or ‘-rc.<number>’ (the - and . are optional). They are considered from old to new in that order with a version number without suffix is considered the newest. This secondary difference is stored in an internal _dev_version attribute.

Two versions are equal if their normal version and dev version are equal. A version is greater if the normal version or dev version is greater. For example:

1.34 < 1.34.1 < 1.35wmf1 < 1.35alpha < 1.35beta1 < 1.35beta2
< 1.35-rc-1 < 1.35-rc.2 < 1.35

Any other suffixes are considered invalid.

Added in version 3.0.

Changed in version 6.1: Dependency of distutils was dropped because the package will be removed with Python 3.12.

Parameters:: version_str (str) – version to parse

MEDIAWIKI_VERSION = re.compile('(\\d+(?:\\.\\d+)+)(-?wmf\\.?(\\d+)|alpha|beta(\\d+)|-?rc\\.?(\\d+)|.*)?')#

static from_generator(generator)[source]#

Create instance from a site’s generator attribute.

Parameters:: generator (str)
Return type:: MediaWikiVersion

class tools.ModuleDeprecationWrapper(module)[source]#

Bases: ModuleType

A wrapper for a module to deprecate classes or variables of it.

Initialise the wrapper.

It will automatically overwrite the module with this instance in sys.modules.

Parameters:: module (types.ModuleType | str) – The module name or instance

add_deprecated_attr(name, replacement=None, *, replacement_name=None, warning_message=None, since='', future_warning=True)[source]#

Add the name to the local deprecated names dict.

Changed in version 7.0: since parameter must be a release number, not a timestamp.

Parameters:

name (str) – The name of the deprecated class or variable. It may not be already deprecated.
replacement (Any) – The replacement value which should be returned instead. If the name is already an attribute of that module this must be None. If None it’ll return the attribute of the module.
replacement_name (str | None) – The name of the new replaced value. Required if replacement is not None and it has no __name__ attribute. If it contains a ‘.’, it will be interpreted as a Python dotted object name, and evaluated when the deprecated object is needed.
warning_message (str | None) – The warning to display, with positional variables: {0} = module, {1} = attribute name, {2} = replacement.
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning

tools.add_decorated_full_name(obj, stacklevel=1)[source]#

Extract full object name, including class, and store in __full_name__.

This must be done on all decorators that are chained together, otherwise the second decorator will have the wrong full name.

Parameters:

obj (object) – An object being decorated
stacklevel (int) – level to use

Return type:

None

tools.add_full_name(obj)[source]#

A decorator to add __full_name__ to the function being decorated.

This should be done for all decorators used in pywikibot, as any decorator that does not add __full_name__ will prevent other decorators in the same chain from being able to obtain it.

This can be used to monkey-patch decorators in other modules. e.g. <xyz>.foo = add_full_name(<xyz>.foo)

Parameters:: obj (callable) – The function to decorate
Returns:: decorating function
Return type:: function

tools.as_filename(string, repl='_')[source]#

Return a string with characters are valid for filenames.

Replace characters that are not possible in file names on some systems, but still are valid in MediaWiki titles:

Unix: /

MediaWiki: /:\\

Windows: /:\\"?*

Spaces are possible on most systems, but are bad for URLs.

Example:

>>> as_filename('How are you?')
'How_are_you_'
>>> as_filename('Say: "Hello"')
'Say___Hello_'
>>> as_filename('foo*bar', '')
'foobar'
>>> as_filename('foo', 'bar')
Traceback (most recent call last):
...
ValueError: Invalid repl parameter 'bar'
>>> as_filename('foo', '?')
Traceback (most recent call last):
...
ValueError: Invalid repl parameter '?'

Added in version 8.0.

Parameters:

string (str) – the string to be modified
repl (str) – the replacement character

Raises:

ValueError – Invalid repl parameter

Return type:

str

tools.cached(*arg)[source]#

Decorator to cache information of an object.

The wrapper adds an attribute to the instance which holds the result of the decorated method. The attribute’s name is the method name with preleading underscore.

Usage:

@cached
def this_method(self)

@cached
def that_method(self, force=False)

No parameter may be used with this decorator. Only a force parameter may be used with the decorated method. All other parameters are discarded and lead to a TypeError.

Note

A property must be decorated on top of the property method below other decorators. This decorator must not be used with functions.

Added in version 7.3.

Raises:: TypeError – decorator must be used without arguments
Parameters:: arg (Callable)
Return type:: Any

class tools.classproperty(cls_method)[source]#

Bases: object

Descriptor class to access a class method as a property.

This class may be used as a decorator:

class Foo:

    _bar = 'baz'  # a class property

    @classproperty
    def bar(cls):  # a class property method
        return cls._bar

Foo.bar gives ‘baz’.

Added in version 3.0.

Hold the class method.

tools.compute_file_hash(filename, sha='sha1', bytes_to_read=None)[source]#

Compute file hash.

Result is expressed as hexdigest().

Added in version 3.0.

Changed in version 8.2: sha may be also a hash constructor, or a callable that returns a hash object.

Parameters:

filename (str | os.PathLike) – filename path
sha (str | Callable[[], Any]) – hash algorithm available with hashlib: sha1(), sha224(), sha256(), sha384(), sha512(), blake2b(), and blake2s(). Additional algorithms like md5(), sha3_224(), sha3_256(), sha3_384(), sha3_512(), shake_128() and shake_256() may also be available. sha must either be a hash algorithm name as a str like 'sha1' (default), a hash constructor like hashlib.sha1, or a callable that returns a hash object like lambda: hashlib.sha1().
bytes_to_read (int | None) – only the first bytes_to_read will be considered; if file size is smaller, the whole file will be considered.

Return type:

str

tools.deprecate_arg(old_arg, new_arg=None)[source]#

Decorator to declare old_arg deprecated and replace it with new_arg.

Usage:

@deprecate_arg('foo', 'bar')
def my_function(bar='baz'): pass
# replaces 'foo' keyword by 'bar' used by my_function

@deprecate_arg('foo', None)
def my_function(): pass
# ignores 'foo' keyword no longer used by my_function

deprecated_args() decorator should be used in favour of this deprecate_arg decorator but it is held to deprecate args of reserved words even for future Python releases and to prevent syntax errors.

Changed in version 9.2: bool type of new_arg is no longer supported.

Parameters:

old_arg (str) – old keyword
new_arg (str | None) – new keyword

tools.deprecate_positionals(since='')[source]#

Decorator for methods that issues warnings for positional arguments.

This decorator allowes positional arguments after keyword-only argument syntax (PEP 3102) but throws a FutureWarning. The decorator makes the needed argument updates before passing them to the called function or method. This decorator may be used for a deprecation period when require keyword-only arguments.

Example

@deprecate_positionals(since='9.2.0')
def f(posarg, *, kwarg):
   ...

f('foo', 'bar')

This function call passes but throws a FutureWarning. Without decorator a TypeError would be raised.

Caution

The decorated function may not use *args or **kwargs. The sequence of keyword-only arguments must match the sequence of the old positional arguments, otherwise the assignment of the arguments to the keyworded arguments will fail.

Added in version 9.2.

Parameters:: since (str) – a version string when some positional arguments were deprecated

tools.deprecated(*args, **kwargs)[source]#

Decorator to output a deprecation warning.

Changed in version 7.0: since keyword must be a release number, not a timestamp.

Keyword Arguments:

instead (str) – if provided, will be used to specify the replacement
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning

tools.deprecated_args(**arg_pairs)[source]#

Decorator to declare multiple args deprecated.

Usage:

@deprecated_args(foo='bar', baz=None)
def my_function(bar='baz'): pass
# replaces 'foo' keyword by 'bar' and ignores 'baz' keyword

Changed in version 3.0.20200703: show a FutureWarning if the arg_pairs value is True; don’t show a warning if the value is an empty string.

Changed in version 6.4: show a FutureWarning for renamed arguments

Changed in version 9.2: bool type argument is no longer supported.

Parameters:: arg_pairs (str | None) – Each entry points to the new argument name. If an argument is to be removed, the value may be either an empty str or None; the later also shows a FutureWarning once.
Raises:: TypeError – arg_pairs value is neither str nor None or

tools.file_mode_checker(filename, mode=384, quiet=False, create=False)[source]#

Check file mode and update it, if needed.

Added in version 3.0.

Parameters:

filename (str) – filename path
mode (int) – requested file mode
quiet (bool) – warn about file mode change if False.
create (bool) – create the file if it does not exist already

Raises:

IOError – The file does not exist and create is False.

tools.first_lower(string)[source]#

Return a string with the first character uncapitalized.

Empty strings are supported. The original string is not changed.

Example:

>>> first_lower('Hello World')
'hello World'

Added in version 3.0.

Parameters:: string (str)
Return type:: str

tools.first_upper(string)[source]#

Return a string with the first character capitalized.

Empty strings are supported. The original string is not changed.

Example:

>>> first_upper('hello World')
'Hello World'

Added in version 3.0.

Note

MediaWiki doesn’t capitalize some characters the same way as Python. This function tries to be close to MediaWiki’s capitalize function in title.php. See T179115 and T200357.

Parameters:: string (str)
Return type:: str

tools.get_wrapper_depth(wrapper)[source]#: Return depth of wrapper function.

Added in version 3.0.

tools.has_module(module, version=None)[source]#

Check if a module can be imported.

Added in version 3.0.

Changed in version 6.1: Dependency of distutils was dropped because the package will be removed with Python 3.12.

Parameters:

module (str)
version (str | None)

Return type:

bool

tools.is_ip_address(value)[source]#

Check if a value is a valid IPv4 or IPv6 address.

Added in version 6.1: Was renamed from is_IP().

Parameters:: value (str) – value to check
Return type:: bool

tools.is_ip_network(value)[source]#

Check if a value is a valid range of IPv4 or IPv6 addresses.

Added in version 9.0.

Parameters:: value (str) – value to check
Return type:: bool

tools.issue_deprecation_warning(name, instead=None, depth=2, *, warning_class=None, since=None)[source]#

Issue a deprecation warning.

Changed in version 7.0: since parameter must be a release number, not a timestamp.

Changed in version 8.2: warning_class and since are keyword-only parameters.

Parameters:

name (str) – the name of the deprecated object
instead (str | None) – suggested replacement for the deprecated object
depth (int) – depth + 1 will be used as stacklevel for the warnings
warning_class (type | None) – a warning class (category) to be used, defaults to FutureWarning
since (str | None) – a version string when the method or function was deprecated

tools.manage_wrapping(wrapper, obj)[source]#

Add attributes to wrapper and wrapped functions.

Added in version 3.0.

Return type:: None

tools.merge_unique_dicts(*args, **kwargs)[source]#

Return a merged dict and make sure that the original dicts keys are unique.

The positional arguments are the dictionaries to be merged. It is also possible to define an additional dict using the keyword arguments.

Added in version 3.0.

tools.normalize_username(username)[source]#

Normalize the username.

Added in version 3.0.

Return type:: str | None

tools.open_archive(filename, mode='rb', use_extension=True)[source]#

Open a file and uncompress it if needed.

This function supports bzip2, gzip, 7zip, lzma, and xz as compression containers. It uses the packages available in the standard library for bzip2, gzip, lzma, and xz so they are always available. 7zip is only available when a 7za program is available and only supports reading from it.

The compression is either selected via the magic number or file ending.

Added in version 3.0.

Parameters:

filename (str) – The filename.
use_extension (bool) – Use the file extension instead of the magic number to determine the type of compression (default True). Must be True when writing or appending.
mode (str) – The mode in which the file should be opened. It may either be ‘r’, ‘rb’, ‘a’, ‘ab’, ‘w’ or ‘wb’. All modes open the file in binary mode. It defaults to ‘rb’.

Raises:

ValueError – When 7za is not available or the opening mode is unknown or it tries to write a 7z archive.
FileNotFoundError – When the filename doesn’t exist and it tries to read from it or it tries to determine the compression algorithm.
OSError – When it’s not a 7z archive but the file extension is 7z. It is also raised by bz2 when its content is invalid. gzip does not immediately raise that error but only on reading it.
lzma.LZMAError – When error occurs during compression or decompression or when initializing the state with lzma or xz.

Returns:

A file-like object returning the uncompressed data in binary mode.

Return type:

file-like object

tools.redirect_func(target, *, source_module=None, target_module=None, old_name=None, class_name=None, since='', future_warning=True)[source]#

Return a function which can be used to redirect to ‘target’.

It also acts like marking that function deprecated and copies all parameters.

Changed in version 7.0: since parameter must be a release number, not a timestamp.

Changed in version 8.2: All parameters except target are keyword-only parameters.

Parameters:

target (callable) – The targeted function which is to be executed.
source_module (str | None) – The module of the old function. If ‘.’ defaults to target_module. If ‘None’ (default) it tries to guess it from the executing function.
target_module (str | None) – The module of the target function. If ‘None’ (default) it tries to get it from the target. Might not work with nested classes.
old_name (str | None) – The old function name. If None it uses the name of the new function.
class_name (str | None) – The name of the class. It’s added to the target and source module (separated by a ‘.’).
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning

Returns:

A new function which adds a warning prior to each execution.

Return type:

callable

tools.remove_last_args(arg_names)[source]#

Decorator to declare all args additionally provided deprecated.

All positional arguments appearing after the normal arguments are marked deprecated. It marks also all keyword arguments present in arg_names as deprecated. Any arguments (positional or keyword) which are not present in arg_names are forwarded. For example a call with 3 parameters and the original function requests one and arg_names contain one name will result in an error, because the function got called with 2 parameters.

The decorated function may not use *args or **kwargs.

Parameters:: arg_names (iterable; for the most explanatory message it should retain the given order (so not a set for example).) – The names of all arguments.

tools.strtobool(val)[source]#

Convert a string representation of truth to True or False.

This is a reimplementation of distutils.util.strtobool due to PEP 632#Migration Advice

Example:

>>> strtobool('yes')
True
>>> strtobool('Off')
False
>>> strtobool('aye')
Traceback (most recent call last):
...
ValueError: invalid truth value 'aye'

Added in version 7.1.

Parameters:: val (str) – True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’.
Raises:: ValueError – val is not a valid truth value
Return type:: bool

class tools.suppress_warnings(message='', category=<class 'Warning'>, filename='')[source]#

Bases: catch_warnings

A decorator/context manager that temporarily suppresses warnings.

Those suppressed warnings that do not match the parameters will be raised shown upon exit.

Added in version 3.0.

Initialize the object.

The parameter semantics are similar to those of warnings.filterwarnings.

Parameters:

message (str) – A string containing a regular expression that the start of the warning message must match. (case-insensitive)
category (type) – A class (a subclass of Warning) of which the warning category must be a subclass in order to match.
filename (str) – A string containing a regular expression that the start of the path to the warning module must match. (case-sensitive)

`tools.deprecate` — Deprecating Decorators and Classes#

Module providing deprecation decorators.

Decorator functions without parameters are _invoked_ differently from decorator functions with function syntax. For example, @deprecated causes a different invocation to @deprecated().

The former is invoked with the decorated function as args[0]. The latter is invoked with the decorator arguments as *args & **kwargs, and it must return a callable which will be invoked with the decorated function as args[0].

The follow deprecators may support both syntax, e.g. @deprecated and @deprecated() both work. In order to achieve that, the code inspects args[0] to see if it callable. Therefore, a decorator must not accept only one arg, and that arg be a callable, as it will be detected as a deprecator without any arguments.

Changed in version 6.4: deprecation decorators moved to _deprecate submodule

class tools._deprecate.ModuleDeprecationWrapper(module)[source]#

Bases: ModuleType

A wrapper for a module to deprecate classes or variables of it.

Initialise the wrapper.

It will automatically overwrite the module with this instance in sys.modules.

Parameters:: module (types.ModuleType | str) – The module name or instance

add_deprecated_attr(name, replacement=None, *, replacement_name=None, warning_message=None, since='', future_warning=True)[source]#

Add the name to the local deprecated names dict.

Changed in version 7.0: since parameter must be a release number, not a timestamp.

Parameters:

name (str) – The name of the deprecated class or variable. It may not be already deprecated.
replacement (Any) – The replacement value which should be returned instead. If the name is already an attribute of that module this must be None. If None it’ll return the attribute of the module.
replacement_name (str | None) – The name of the new replaced value. Required if replacement is not None and it has no __name__ attribute. If it contains a ‘.’, it will be interpreted as a Python dotted object name, and evaluated when the deprecated object is needed.
warning_message (str | None) – The warning to display, with positional variables: {0} = module, {1} = attribute name, {2} = replacement.
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning

tools._deprecate.add_decorated_full_name(obj, stacklevel=1)[source]#

Extract full object name, including class, and store in __full_name__.

This must be done on all decorators that are chained together, otherwise the second decorator will have the wrong full name.

Parameters:

obj (object) – An object being decorated
stacklevel (int) – level to use

Return type:

None

tools._deprecate.add_full_name(obj)[source]#

A decorator to add __full_name__ to the function being decorated.

This should be done for all decorators used in pywikibot, as any decorator that does not add __full_name__ will prevent other decorators in the same chain from being able to obtain it.

This can be used to monkey-patch decorators in other modules. e.g. <xyz>.foo = add_full_name(<xyz>.foo)

Parameters:: obj (callable) – The function to decorate
Returns:: decorating function
Return type:: function

tools._deprecate.deprecate_arg(old_arg, new_arg=None)[source]#

Decorator to declare old_arg deprecated and replace it with new_arg.

Usage:

@deprecate_arg('foo', 'bar')
def my_function(bar='baz'): pass
# replaces 'foo' keyword by 'bar' used by my_function

@deprecate_arg('foo', None)
def my_function(): pass
# ignores 'foo' keyword no longer used by my_function

Changed in version 9.2: bool type of new_arg is no longer supported.

Parameters:

old_arg (str) – old keyword
new_arg (str | None) – new keyword

tools._deprecate.deprecate_positionals(since='')[source]#

Decorator for methods that issues warnings for positional arguments.

Example

@deprecate_positionals(since='9.2.0')
def f(posarg, *, kwarg):
   ...

f('foo', 'bar')

This function call passes but throws a FutureWarning. Without decorator a TypeError would be raised.

Caution

Added in version 9.2.

Parameters:: since (str) – a version string when some positional arguments were deprecated

tools._deprecate.deprecated(*args, **kwargs)[source]#

Decorator to output a deprecation warning.

Changed in version 7.0: since keyword must be a release number, not a timestamp.

Keyword Arguments:

instead (str) – if provided, will be used to specify the replacement
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning

tools._deprecate.deprecated_args(**arg_pairs)[source]#

Decorator to declare multiple args deprecated.

Usage:

@deprecated_args(foo='bar', baz=None)
def my_function(bar='baz'): pass
# replaces 'foo' keyword by 'bar' and ignores 'baz' keyword

Changed in version 3.0.20200703: show a FutureWarning if the arg_pairs value is True; don’t show a warning if the value is an empty string.

Changed in version 6.4: show a FutureWarning for renamed arguments

Changed in version 9.2: bool type argument is no longer supported.

Parameters:: arg_pairs (str | None) – Each entry points to the new argument name. If an argument is to be removed, the value may be either an empty str or None; the later also shows a FutureWarning once.
Raises:: TypeError – arg_pairs value is neither str nor None or

tools._deprecate.get_wrapper_depth(wrapper)[source]#: Return depth of wrapper function.

Added in version 3.0.

tools._deprecate.issue_deprecation_warning(name, instead=None, depth=2, *, warning_class=None, since=None)[source]#

Issue a deprecation warning.

Changed in version 7.0: since parameter must be a release number, not a timestamp.

Changed in version 8.2: warning_class and since are keyword-only parameters.

Parameters:

name (str) – the name of the deprecated object
instead (str | None) – suggested replacement for the deprecated object
depth (int) – depth + 1 will be used as stacklevel for the warnings
warning_class (type | None) – a warning class (category) to be used, defaults to FutureWarning
since (str | None) – a version string when the method or function was deprecated

tools._deprecate.manage_wrapping(wrapper, obj)[source]#

Add attributes to wrapper and wrapped functions.

Added in version 3.0.

Return type:: None

tools._deprecate.redirect_func(target, *, source_module=None, target_module=None, old_name=None, class_name=None, since='', future_warning=True)[source]#

Return a function which can be used to redirect to ‘target’.

It also acts like marking that function deprecated and copies all parameters.

Changed in version 7.0: since parameter must be a release number, not a timestamp.

Changed in version 8.2: All parameters except target are keyword-only parameters.

Parameters:

target (callable) – The targeted function which is to be executed.
source_module (str | None) – The module of the old function. If ‘.’ defaults to target_module. If ‘None’ (default) it tries to guess it from the executing function.
target_module (str | None) – The module of the target function. If ‘None’ (default) it tries to get it from the target. Might not work with nested classes.
old_name (str | None) – The old function name. If None it uses the name of the new function.
class_name (str | None) – The name of the class. It’s added to the target and source module (separated by a ‘.’).
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning

Returns:

A new function which adds a warning prior to each execution.

Return type:

callable

tools._deprecate.remove_last_args(arg_names)[source]#

Decorator to declare all args additionally provided deprecated.

The decorated function may not use *args or **kwargs.

Parameters:: arg_names (iterable; for the most explanatory message it should retain the given order (so not a set for example).) – The names of all arguments.

`tools._logging` — logging.Formatter Subclass#

Logging tools.

class tools._logging.LoggingFormatter(fmt=None, datefmt=None, style='%', validate=True)[source]#

Bases: Formatter

Format LogRecords for output to file.

Initialize the formatter with specified format strings.

Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format.

Use a style parameter of ‘%’, ‘{’ or ‘$’ to specify that you want to use one of %-formatting, str.format() ({}) formatting or string.Template formatting in your format string.

Changed in version 3.2: Added the style parameter.

format(record)[source]#: Strip trailing newlines before outputting text to file.

tools — Miscellaneous Helper Functions#

tools.deprecate — Deprecating Decorators and Classes#

tools._logging — logging.Formatter Subclass#

`tools` — Miscellaneous Helper Functions#

`tools.deprecate` — Deprecating Decorators and Classes#

`tools._logging` — logging.Formatter Subclass#