tools
— Miscellaneous Helper Functions#
Miscellaneous helper functions (not wiki-dependent).
Deprecated since version 7.6: The CombinedError, DequeGenerator, EmptyDefault, EMPTY_DEFAULT
and SizedKeyCollection objects. Import them from
tools.collections
instead.
Deprecated since version 7.6: The itergroup function. Use backports.batched()
or
itertools.batched
instead.
Deprecated since version 7.6: The filter_unique, intersect_generators, islice_with_ellipsis
and roundrobin_generators functions. Import them from
tools.itertools
instead.
Deprecated since version 7.7: The RLock, ThreadedGenerator and ThreadList classes.
Import them from tools.threading
instead.
- class tools.ComparableMixin[source]#
Bases:
ABC
Mixin class to allow comparing to other objects which are comparable.
Added in version 3.0.
- class tools.MediaWikiVersion(version_str)[source]#
Bases:
object
Version object to allow comparing ‘wmf’ versions with normal ones.
The version mainly consist of digits separated by periods. After that is a suffix which may only be ‘wmf<number>’, ‘alpha’, ‘beta<number>’ or ‘-rc.<number>’ (the - and . are optional). They are considered from old to new in that order with a version number without suffix is considered the newest. This secondary difference is stored in an internal _dev_version attribute.
Two versions are equal if their normal version and dev version are equal. A version is greater if the normal version or dev version is greater. For example:
1.34 < 1.34.1 < 1.35wmf1 < 1.35alpha < 1.35beta1 < 1.35beta2 < 1.35-rc-1 < 1.35-rc.2 < 1.35
Any other suffixes are considered invalid.
Added in version 3.0.
Changed in version 6.1: Dependency of distutils was dropped because the package will be removed with Python 3.12.
- Parameters:
version_str (str) – version to parse
- MEDIAWIKI_VERSION = re.compile('(\\d+(?:\\.\\d+)+)(-?wmf\\.?(\\d+)|alpha|beta(\\d+)|-?rc\\.?(\\d+)|.*)?')#
- class tools.ModuleDeprecationWrapper(module)[source]#
Bases:
ModuleType
A wrapper for a module to deprecate classes or variables of it.
Initialise the wrapper.
It will automatically overwrite the module with this instance in
sys.modules
.- Parameters:
module (types.ModuleType | str) – The module name or instance
- add_deprecated_attr(name, replacement=None, *, replacement_name=None, warning_message=None, since='', future_warning=True)[source]#
Add the name to the local deprecated names dict.
Changed in version 7.0:
since
parameter must be a release number, not a timestamp.- Parameters:
name (str) – The name of the deprecated class or variable. It may not be already deprecated.
replacement (Any) – The replacement value which should be returned instead. If the name is already an attribute of that module this must be None. If None it’ll return the attribute of the module.
replacement_name (str | None) – The name of the new replaced value. Required if
replacement
is not None and it has no __name__ attribute. If it contains a ‘.’, it will be interpreted as a Python dotted object name, and evaluated when the deprecated object is needed.warning_message (str | None) – The warning to display, with positional variables: {0} = module, {1} = attribute name, {2} = replacement.
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning
- tools.add_decorated_full_name(obj, stacklevel=1)[source]#
Extract full object name, including class, and store in __full_name__.
This must be done on all decorators that are chained together, otherwise the second decorator will have the wrong full name.
- Parameters:
obj (object) – An object being decorated
stacklevel (int) – level to use
- Return type:
None
- tools.add_full_name(obj)[source]#
A decorator to add __full_name__ to the function being decorated.
This should be done for all decorators used in pywikibot, as any decorator that does not add __full_name__ will prevent other decorators in the same chain from being able to obtain it.
This can be used to monkey-patch decorators in other modules. e.g. <xyz>.foo = add_full_name(<xyz>.foo)
- Parameters:
obj (callable) – The function to decorate
- Returns:
decorating function
- Return type:
function
- tools.as_filename(string, repl='_')[source]#
Return a string with characters are valid for filenames.
Replace characters that are not possible in file names on some systems, but still are valid in MediaWiki titles:
Unix:
/
MediaWiki:
/:\\
Windows:
/:\\"?*
Spaces are possible on most systems, but are bad for URLs.
Example:
>>> as_filename('How are you?') 'How_are_you_' >>> as_filename('Say: "Hello"') 'Say___Hello_' >>> as_filename('foo*bar', '') 'foobar' >>> as_filename('foo', 'bar') Traceback (most recent call last): ... ValueError: Invalid repl parameter 'bar' >>> as_filename('foo', '?') Traceback (most recent call last): ... ValueError: Invalid repl parameter '?'
Added in version 8.0.
- Parameters:
string (str) – the string to be modified
repl (str) – the replacement character
- Raises:
ValueError – Invalid repl parameter
- Return type:
str
- tools.cached(*arg)[source]#
Decorator to cache information of an object.
The wrapper adds an attribute to the instance which holds the result of the decorated method. The attribute’s name is the method name with preleading underscore.
Usage:
@cached def this_method(self) @cached def that_method(self, force=False)
No parameter may be used with this decorator. Only a force parameter may be used with the decorated method. All other parameters are discarded and lead to a TypeError.
Note
A property must be decorated on top of the property method below other decorators. This decorator must not be used with functions.
Added in version 7.3.
- Raises:
TypeError – decorator must be used without arguments
- Parameters:
arg (Callable)
- Return type:
Any
- class tools.classproperty(cls_method)[source]#
Bases:
object
Descriptor class to access a class method as a property.
This class may be used as a decorator:
class Foo: _bar = 'baz' # a class property @classproperty def bar(cls): # a class property method return cls._bar
Foo.bar gives ‘baz’.
Added in version 3.0.
Hold the class method.
- tools.compute_file_hash(filename, sha='sha1', bytes_to_read=None)[source]#
Compute file hash.
Result is expressed as hexdigest().
Added in version 3.0.
Changed in version 8.2: sha may be also a hash constructor, or a callable that returns a hash object.
- Parameters:
filename (str | PathLike) – filename path
sha (str | Callable[[], Any]) – hash algorithm available with hashlib:
sha1()
,sha224()
,sha256()
,sha384()
,sha512()
,blake2b()
, andblake2s()
. Additional algorithms likemd5()
,sha3_224()
,sha3_256()
,sha3_384()
,sha3_512()
,shake_128()
andshake_256()
may also be available. sha must either be a hash algorithm name as a str like'sha1'
(default), a hash constructor likehashlib.sha1
, or a callable that returns a hash object likelambda: hashlib.sha1()
.bytes_to_read (int | None) – only the first bytes_to_read will be considered; if file size is smaller, the whole file will be considered.
- Return type:
str
- tools.deprecate_arg(old_arg, new_arg=None)[source]#
Decorator to declare old_arg deprecated and replace it with new_arg.
Usage:
@deprecate_arg('foo', 'bar') def my_function(bar='baz'): pass # replaces 'foo' keyword by 'bar' used by my_function @deprecate_arg('foo', None) def my_function(): pass # ignores 'foo' keyword no longer used by my_function
deprecated_args()
decorator should be used in favour of thisdeprecate_arg
decorator but it is held to deprecate args of reserved words even for future Python releases and to prevent syntax errors.Changed in version 9.2: bool type of new_arg is no longer supported.
- Parameters:
old_arg (str) – old keyword
new_arg (str | None) – new keyword
- tools.deprecate_positionals(since='')[source]#
Decorator for methods that issues warnings for positional arguments.
This decorator allowes positional arguments after keyword-only argument syntax (PEP 3102) but throws a FutureWarning. The decorator makes the needed argument updates before passing them to the called function or method. This decorator may be used for a deprecation period when require keyword-only arguments.
Example
@deprecate_positionals(since='9.2.0') def f(posarg, *, kwarg): ... f('foo', 'bar')
This function call passes but throws a FutureWarning. Without decorator a TypeError would be raised.
Caution
The decorated function may not use
*args
or**kwargs
. The sequence of keyword-only arguments must match the sequence of the old positional arguments, otherwise the assignment of the arguments to the keyworded arguments will fail.Added in version 9.2.
- Parameters:
since (str) – a version string when some positional arguments were deprecated
- tools.deprecated(*args, **kwargs)[source]#
Decorator to output a deprecation warning.
Changed in version 7.0:
since
keyword must be a release number, not a timestamp.- Keyword Arguments:
instead (str) – if provided, will be used to specify the replacement
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning
- tools.deprecated_args(**arg_pairs)[source]#
Decorator to declare multiple args deprecated.
Usage:
@deprecated_args(foo='bar', baz=None) def my_function(bar='baz'): pass # replaces 'foo' keyword by 'bar' and ignores 'baz' keyword
Changed in version 3.0.20200703: show a FutureWarning if the arg_pairs value is True; don’t show a warning if the value is an empty string.
Changed in version 6.4: show a FutureWarning for renamed arguments
Changed in version 9.2: bool type argument is no longer supported.
- Parameters:
arg_pairs (str | None) – Each entry points to the new argument name. If an argument is to be removed, the value may be either an empty str or
None
; the later also shows aFutureWarning
once.- Raises:
TypeError – arg_pairs value is neither str nor None or
- tools.file_mode_checker(filename, mode=384, quiet=False, create=False)[source]#
Check file mode and update it, if needed.
Added in version 3.0.
- Parameters:
filename (str) – filename path
mode (int) – requested file mode
quiet (bool) – warn about file mode change if False.
create (bool) – create the file if it does not exist already
- Raises:
IOError – The file does not exist and
create
is False.
- tools.first_lower(string)[source]#
Return a string with the first character uncapitalized.
Empty strings are supported. The original string is not changed.
Example:
>>> first_lower('Hello World') 'hello World'
Added in version 3.0.
- Parameters:
string (str)
- Return type:
str
- tools.first_upper(string)[source]#
Return a string with the first character capitalized.
Empty strings are supported. The original string is not changed.
Example:
>>> first_upper('hello World') 'Hello World'
Added in version 3.0.
Note
MediaWiki doesn’t capitalize some characters the same way as Python. This function tries to be close to MediaWiki’s capitalize function in title.php. See T179115 and T200357.
- Parameters:
string (str)
- Return type:
str
- tools.has_module(module, version=None)[source]#
Check if a module can be imported.
Added in version 3.0.
Changed in version 6.1: Dependency of distutils was dropped because the package will be removed with Python 3.12.
- Parameters:
module (str)
version (str | None)
- Return type:
bool
- tools.is_ip_address(value)[source]#
Check if a value is a valid IPv4 or IPv6 address.
Added in version 6.1: Was renamed from
is_IP()
.- Parameters:
value (str) – value to check
- Return type:
bool
- tools.is_ip_network(value)[source]#
Check if a value is a valid range of IPv4 or IPv6 addresses.
Added in version 9.0.
- Parameters:
value (str) – value to check
- Return type:
bool
- tools.issue_deprecation_warning(name, instead=None, depth=2, *, warning_class=None, since=None)[source]#
Issue a deprecation warning.
Changed in version 7.0: since parameter must be a release number, not a timestamp.
Changed in version 8.2: warning_class and since are keyword-only parameters.
- Parameters:
name (str) – the name of the deprecated object
instead (str | None) – suggested replacement for the deprecated object
depth (int) – depth + 1 will be used as stacklevel for the warnings
warning_class (type | None) – a warning class (category) to be used, defaults to FutureWarning
since (str | None) – a version string when the method or function was deprecated
- tools.manage_wrapping(wrapper, obj)[source]#
Add attributes to wrapper and wrapped functions.
Added in version 3.0.
- Return type:
None
- tools.merge_unique_dicts(*args, **kwargs)[source]#
Return a merged dict and make sure that the original dicts keys are unique.
The positional arguments are the dictionaries to be merged. It is also possible to define an additional dict using the keyword arguments.
Added in version 3.0.
- tools.normalize_username(username)[source]#
Normalize the username.
Added in version 3.0.
- Return type:
str | None
- tools.open_archive(filename, mode='rb', use_extension=True)[source]#
Open a file and uncompress it if needed.
This function supports bzip2, gzip, 7zip, lzma, and xz as compression containers. It uses the packages available in the standard library for bzip2, gzip, lzma, and xz so they are always available. 7zip is only available when a 7za program is available and only supports reading from it.
The compression is either selected via the magic number or file ending.
Added in version 3.0.
- Parameters:
filename (str) – The filename.
use_extension (bool) – Use the file extension instead of the magic number to determine the type of compression (default True). Must be True when writing or appending.
mode (str) – The mode in which the file should be opened. It may either be ‘r’, ‘rb’, ‘a’, ‘ab’, ‘w’ or ‘wb’. All modes open the file in binary mode. It defaults to ‘rb’.
- Raises:
ValueError – When 7za is not available or the opening mode is unknown or it tries to write a 7z archive.
FileNotFoundError – When the filename doesn’t exist and it tries to read from it or it tries to determine the compression algorithm.
OSError – When it’s not a 7z archive but the file extension is 7z. It is also raised by bz2 when its content is invalid. gzip does not immediately raise that error but only on reading it.
lzma.LZMAError – When error occurs during compression or decompression or when initializing the state with lzma or xz.
- Returns:
A file-like object returning the uncompressed data in binary mode.
- Return type:
file-like object
- tools.redirect_func(target, *, source_module=None, target_module=None, old_name=None, class_name=None, since='', future_warning=True)[source]#
Return a function which can be used to redirect to ‘target’.
It also acts like marking that function deprecated and copies all parameters.
Changed in version 7.0: since parameter must be a release number, not a timestamp.
Changed in version 8.2: All parameters except target are keyword-only parameters.
- Parameters:
target (callable) – The targeted function which is to be executed.
source_module (str | None) – The module of the old function. If ‘.’ defaults to target_module. If ‘None’ (default) it tries to guess it from the executing function.
target_module (str | None) – The module of the target function. If ‘None’ (default) it tries to get it from the target. Might not work with nested classes.
old_name (str | None) – The old function name. If None it uses the name of the new function.
class_name (str | None) – The name of the class. It’s added to the target and source module (separated by a ‘.’).
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning
- Returns:
A new function which adds a warning prior to each execution.
- Return type:
callable
- tools.remove_last_args(arg_names)[source]#
Decorator to declare all args additionally provided deprecated.
All positional arguments appearing after the normal arguments are marked deprecated. It marks also all keyword arguments present in arg_names as deprecated. Any arguments (positional or keyword) which are not present in arg_names are forwarded. For example a call with 3 parameters and the original function requests one and arg_names contain one name will result in an error, because the function got called with 2 parameters.
The decorated function may not use
*args
or**kwargs
.- Parameters:
arg_names (iterable; for the most explanatory message it should retain the given order (so not a set for example).) – The names of all arguments.
- tools.strtobool(val)[source]#
Convert a string representation of truth to True or False.
This is a reimplementation of distutils.util.strtobool due to PEP 632#Migration Advice
Example:
>>> strtobool('yes') True >>> strtobool('Off') False >>> strtobool('aye') Traceback (most recent call last): ... ValueError: invalid truth value 'aye'
Added in version 7.1.
- Parameters:
val (str) – True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’.
- Raises:
ValueError –
val
is not a valid truth value- Return type:
bool
- class tools.suppress_warnings(message='', category=<class 'Warning'>, filename='')[source]#
Bases:
catch_warnings
A decorator/context manager that temporarily suppresses warnings.
Those suppressed warnings that do not match the parameters will be raised shown upon exit.
Added in version 3.0.
Initialize the object.
The parameter semantics are similar to those of
warnings.filterwarnings
.- Parameters:
message (str) – A string containing a regular expression that the start of the warning message must match. (case-insensitive)
category (type) – A class (a subclass of Warning) of which the warning category must be a subclass in order to match.
filename (str) – A string containing a regular expression that the start of the path to the warning module must match. (case-sensitive)
tools.deprecate
— Deprecating Decorators and Classes#
Module providing deprecation decorators.
Decorator functions without parameters are _invoked_ differently from decorator functions with function syntax. For example, @deprecated causes a different invocation to @deprecated().
The former is invoked with the decorated function as args[0].
The latter is invoked with the decorator arguments as *args
&
**kwargs
, and it must return a callable which will be invoked with
the decorated function as args[0].
The follow deprecators may support both syntax, e.g. @deprecated and @deprecated() both work. In order to achieve that, the code inspects args[0] to see if it callable. Therefore, a decorator must not accept only one arg, and that arg be a callable, as it will be detected as a deprecator without any arguments.
Changed in version 6.4: deprecation decorators moved to _deprecate submodule
- class tools._deprecate.ModuleDeprecationWrapper(module)[source]#
Bases:
ModuleType
A wrapper for a module to deprecate classes or variables of it.
Initialise the wrapper.
It will automatically overwrite the module with this instance in
sys.modules
.- Parameters:
module (types.ModuleType | str) – The module name or instance
- add_deprecated_attr(name, replacement=None, *, replacement_name=None, warning_message=None, since='', future_warning=True)[source]#
Add the name to the local deprecated names dict.
Changed in version 7.0:
since
parameter must be a release number, not a timestamp.- Parameters:
name (str) – The name of the deprecated class or variable. It may not be already deprecated.
replacement (Any) – The replacement value which should be returned instead. If the name is already an attribute of that module this must be None. If None it’ll return the attribute of the module.
replacement_name (str | None) – The name of the new replaced value. Required if
replacement
is not None and it has no __name__ attribute. If it contains a ‘.’, it will be interpreted as a Python dotted object name, and evaluated when the deprecated object is needed.warning_message (str | None) – The warning to display, with positional variables: {0} = module, {1} = attribute name, {2} = replacement.
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning
- tools._deprecate.add_decorated_full_name(obj, stacklevel=1)[source]#
Extract full object name, including class, and store in __full_name__.
This must be done on all decorators that are chained together, otherwise the second decorator will have the wrong full name.
- Parameters:
obj (object) – An object being decorated
stacklevel (int) – level to use
- Return type:
None
- tools._deprecate.add_full_name(obj)[source]#
A decorator to add __full_name__ to the function being decorated.
This should be done for all decorators used in pywikibot, as any decorator that does not add __full_name__ will prevent other decorators in the same chain from being able to obtain it.
This can be used to monkey-patch decorators in other modules. e.g. <xyz>.foo = add_full_name(<xyz>.foo)
- Parameters:
obj (callable) – The function to decorate
- Returns:
decorating function
- Return type:
function
- tools._deprecate.deprecate_arg(old_arg, new_arg=None)[source]#
Decorator to declare old_arg deprecated and replace it with new_arg.
Usage:
@deprecate_arg('foo', 'bar') def my_function(bar='baz'): pass # replaces 'foo' keyword by 'bar' used by my_function @deprecate_arg('foo', None) def my_function(): pass # ignores 'foo' keyword no longer used by my_function
deprecated_args()
decorator should be used in favour of thisdeprecate_arg
decorator but it is held to deprecate args of reserved words even for future Python releases and to prevent syntax errors.Changed in version 9.2: bool type of new_arg is no longer supported.
- Parameters:
old_arg (str) – old keyword
new_arg (str | None) – new keyword
- tools._deprecate.deprecate_positionals(since='')[source]#
Decorator for methods that issues warnings for positional arguments.
This decorator allowes positional arguments after keyword-only argument syntax (PEP 3102) but throws a FutureWarning. The decorator makes the needed argument updates before passing them to the called function or method. This decorator may be used for a deprecation period when require keyword-only arguments.
Example
@deprecate_positionals(since='9.2.0') def f(posarg, *, kwarg): ... f('foo', 'bar')
This function call passes but throws a FutureWarning. Without decorator a TypeError would be raised.
Caution
The decorated function may not use
*args
or**kwargs
. The sequence of keyword-only arguments must match the sequence of the old positional arguments, otherwise the assignment of the arguments to the keyworded arguments will fail.Added in version 9.2.
- Parameters:
since (str) – a version string when some positional arguments were deprecated
- tools._deprecate.deprecated(*args, **kwargs)[source]#
Decorator to output a deprecation warning.
Changed in version 7.0:
since
keyword must be a release number, not a timestamp.- Keyword Arguments:
instead (str) – if provided, will be used to specify the replacement
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning
- tools._deprecate.deprecated_args(**arg_pairs)[source]#
Decorator to declare multiple args deprecated.
Usage:
@deprecated_args(foo='bar', baz=None) def my_function(bar='baz'): pass # replaces 'foo' keyword by 'bar' and ignores 'baz' keyword
Changed in version 3.0.20200703: show a FutureWarning if the arg_pairs value is True; don’t show a warning if the value is an empty string.
Changed in version 6.4: show a FutureWarning for renamed arguments
Changed in version 9.2: bool type argument is no longer supported.
- Parameters:
arg_pairs (str | None) – Each entry points to the new argument name. If an argument is to be removed, the value may be either an empty str or
None
; the later also shows aFutureWarning
once.- Raises:
TypeError – arg_pairs value is neither str nor None or
- tools._deprecate.get_wrapper_depth(wrapper)[source]#
Return depth of wrapper function.
Added in version 3.0.
- tools._deprecate.issue_deprecation_warning(name, instead=None, depth=2, *, warning_class=None, since=None)[source]#
Issue a deprecation warning.
Changed in version 7.0: since parameter must be a release number, not a timestamp.
Changed in version 8.2: warning_class and since are keyword-only parameters.
- Parameters:
name (str) – the name of the deprecated object
instead (str | None) – suggested replacement for the deprecated object
depth (int) – depth + 1 will be used as stacklevel for the warnings
warning_class (type | None) – a warning class (category) to be used, defaults to FutureWarning
since (str | None) – a version string when the method or function was deprecated
- tools._deprecate.manage_wrapping(wrapper, obj)[source]#
Add attributes to wrapper and wrapped functions.
Added in version 3.0.
- Return type:
None
- tools._deprecate.redirect_func(target, *, source_module=None, target_module=None, old_name=None, class_name=None, since='', future_warning=True)[source]#
Return a function which can be used to redirect to ‘target’.
It also acts like marking that function deprecated and copies all parameters.
Changed in version 7.0: since parameter must be a release number, not a timestamp.
Changed in version 8.2: All parameters except target are keyword-only parameters.
- Parameters:
target (callable) – The targeted function which is to be executed.
source_module (str | None) – The module of the old function. If ‘.’ defaults to target_module. If ‘None’ (default) it tries to guess it from the executing function.
target_module (str | None) – The module of the target function. If ‘None’ (default) it tries to get it from the target. Might not work with nested classes.
old_name (str | None) – The old function name. If None it uses the name of the new function.
class_name (str | None) – The name of the class. It’s added to the target and source module (separated by a ‘.’).
since (str) – a version string when the method or function was deprecated
future_warning (bool) – if True a FutureWarning will be thrown, otherwise it provides a DeprecationWarning
- Returns:
A new function which adds a warning prior to each execution.
- Return type:
callable
- tools._deprecate.remove_last_args(arg_names)[source]#
Decorator to declare all args additionally provided deprecated.
All positional arguments appearing after the normal arguments are marked deprecated. It marks also all keyword arguments present in arg_names as deprecated. Any arguments (positional or keyword) which are not present in arg_names are forwarded. For example a call with 3 parameters and the original function requests one and arg_names contain one name will result in an error, because the function got called with 2 parameters.
The decorated function may not use
*args
or**kwargs
.- Parameters:
arg_names (iterable; for the most explanatory message it should retain the given order (so not a set for example).) – The names of all arguments.
tools._logging
— logging.Formatter Subclass#
Logging tools.
- class tools._logging.LoggingFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)[source]#
Bases:
Formatter
Format LogRecords for output to file.
Initialize the formatter with specified format strings.
Initialize the formatter either with the specified format string, or a default as described above. Allow for specialized date formatting with the optional datefmt argument. If datefmt is omitted, you get an ISO8601-like (or RFC 3339-like) format.
Use a style parameter of ‘%’, ‘{’ or ‘$’ to specify that you want to use one of %-formatting,
str.format()
({}
) formatting orstring.Template
formatting in your format string.Changed in version 3.2: Added the
style
parameter.