throttle — Slow Down Wiki I/O#

Mechanisms to regulate the read and write rate to wiki servers.

This module defines the Throttle class, which ensures that automated access to wiki servers adheres to responsible rate limits. It avoids overloading the servers by introducing configurable delays between requests, and coordinates these limits across processes using a shared control file throttle.ctrl.

It supports both read and write throttling, automatic adjustment based on the number of concurrent bot instances, and optional lag-aware delays.

class throttle.ProcEntry(module_id, pid, time, site)[source]#

Bases: NamedTuple

ProcEntry namedtuple.

Create new instance of ProcEntry(module_id, pid, time, site)

Parameters:

  • module_id (str)

  • pid (int)

  • time (int)

  • site (str)

module_id: str#

Alias for field number 0

pid: int#

Alias for field number 1

site: str#

Alias for field number 3

time: int#

Alias for field number 2

class throttle.Throttle(site, *, mindelay=None, maxdelay=None, writedelay=None)[source]#

Bases: object

Control rate of access to wiki server.

Calling this object blocks the calling thread until at least 'delay' seconds have passed since the previous call.

Each Site initiates one Throttle object (site.throttle) to control the rate of access.

Parameters:

  • site (pywikibot.site.BaseSite | str) – site or sitename for this Throttle. If site is an empty string, it will not be written to the throttle.ctrl file.

  • mindelay (int | None) – The minimal delay, also used for read access

  • maxdelay (int | None) – The maximal delay

  • writedelay (int | float | None) – The write delay

checkMultiplicity()[source]#

Count running processes for site and set process_multiplicity.

Changed in version 7.0: process is not written to throttle.ctrl file if site is empty.

Return type:

None

checkdelay: int = 300#
drop()[source]#

Remove me from the list of running bot processes.

Return type:

None

property dropdelay#

Ignore processes that have not made a check in this many seconds.

Deprecated since version 8.4: use expiry instead.

expiry: int = 600#
getDelay(write=False)[source]#

Return the current delay, adjusted for active processes.

Deprecated since version 10.3.0: Use get_delay() instead.

Parameters:

write (bool)

Return type:

float

get_delay(*, write=False)[source]#

Return the current delay, adjusted for active processes.

Compute the delay for a read or write operation, factoring in process concurrency. This method does not account for how much time has already passed since the last access — use waittime() for that.

Added in version 10.3.0: Renamed from getDelay().

Parameters:

write (bool) – Whether the operation is a write (uses writedelay).

Returns:

The delay in seconds before the next operation should occur.

Return type:

float

get_pid(module)[source]#

Get the global pid if the module is running multiple times.

Parameters:

module (str)

Return type:

int

lag(lagtime=None)[source]#

Seize the throttle lock due to server lag.

Usually the self.retry-after value from response_header of the last request if available which will be used for wait time. Otherwise lagtime from api maxlag is used. If neither self.retry_after nor lagtime is set, fallback to config.retry_wait.

If the lagtime is disproportionately high compared to self.retry_after value, the wait time will be increased.

This method is used by api.request. It will prevent any thread from accessing this site.

Parameters:

lagtime (float | None) – The time to wait for the next request which is the last maxlag time from api warning. This is only used as a fallback if self.retry_after isn’t set.

Return type:

None

property next_multiplicity: float#

Factor to scale delay time based on upcoming request size.

Deprecated since version 10.3.0.

property releasepid#

Free the process id after this many seconds.

Deprecated since version 8.4: use expiry instead.

setDelays(delay=None, writedelay=None, absolute=False)[source]#

Set the nominal delays in seconds.

Deprecated since version 10.3.0: Use set_delays() instead.

Parameters:

absolute (bool)

Return type:

None

set_delays(*, delay=None, writedelay=None, absolute=False)[source]#

Set the nominal delays in seconds.

Defaults to config values.

Added in version 10.3.0: Renamed from setDelays().

Parameters:

absolute (bool)

Return type:

None

static wait(seconds)[source]#

Wait for seconds seconds.

Announce the delay if it exceeds a preset limit.

Parameters:

seconds (int | float)

Return type:

None

waittime(write=False)[source]#

Return waiting time in seconds.

The result is for a query that would be made right now.

Parameters:

write (bool)

throttle.pid: bool | int = False#

Global process identifier.

When the first Throttle is instantiated, it will set this variable to a positive integer, which will apply to all throttle objects created by this process.