throttle — Slow Down Wiki I/O#

Mechanisms to regulate the read and write rate to wiki servers.

This module defines the Throttle class, which ensures that automated access to wiki servers adheres to responsible rate limits. It avoids overloading the servers by introducing configurable delays between requests, and coordinates these limits across processes using a shared control file throttle.ctrl.

It supports both read and write throttling, automatic adjustment based on the number of concurrent bot instances, and optional lag-aware delays.

class throttle.ProcEntry(module_id, pid, time, site)[source]#

Bases: NamedTuple

ProcEntry namedtuple.

Create new instance of ProcEntry(module_id, pid, time, site)

Parameters:
  • module_id (str)

  • pid (int)

  • time (int)

  • site (str)

module_id: str#

Alias for field number 0

pid: int#

Alias for field number 1

site: str#

Alias for field number 3

time: int#

Alias for field number 2

class throttle.Throttle(site, *, mindelay=None, maxdelay=None, writedelay=None)[source]#

Bases: object

Control rate of access to wiki server.

Calling this object blocks the calling thread until at least 'delay' seconds have passed since the previous call.

Each Site initiates one Throttle object (site.throttle) to control the rate of access.

Parameters:
  • site (pywikibot.site.BaseSite | str) – site or sitename for this Throttle. If site is an empty string, it will not be written to the throttle.ctrl file.

  • mindelay (int | None) – The minimal delay, also used for read access

  • maxdelay (int | None) – The maximal delay

  • writedelay (int | float | None) – The write delay

checkMultiplicity()[source]#

Count running processes for site and set process_multiplicity.

Changed in version 7.0: process is not written to throttle.ctrl file if site is empty.

Return type:

None

checkdelay: int = 300#
delay: int#

the minimum access delay, usually set by config.minthrottle

drop()[source]#

Remove me from the list of running bot processes.

Return type:

None

expiry: int = 600#
getDelay(write=False)[source]#

Deprecated; use get_delay instead.

Return the current delay, adjusted for active processes.

Deprecated since version 10.3.0: Use get_delay() instead.

Parameters:

write (bool)

Return type:

float

get_delay(*, write=False)[source]#

Return the current delay, adjusted for active processes.

Computes the delay for a read or write operation, taking into account process concurrency and any pending retry-after value. The returned value already includes the process_multiplicity factor. This method does not consider how much time has passed since the last access — use waittime() for that.

Added in version 10.3.0: Renamed from getDelay().

Changed in version 11.0: The delay now takes any pending retry_after into account.

Parameters:

write (bool) – Whether the operation is a write (uses writedelay).

Returns:

The delay in seconds before the next operation should occur.

Return type:

float

get_pid(module)[source]#

Get the global pid if the module is running multiple times.

Parameters:

module (str)

Return type:

int

lag(lagtime=None)[source]#

Seize the throttle lock due to server lag.

The wait time is determined by the lagtime parameter, which usually contains the lag reported by the API in a maxlag error. If lagtime is not set, config.retry_wait is used instead.

If self.retry_after is set, the wait time is at least that value, as recommended by the server. The actual wait time is capped at config.retry_max seconds.

This method is used by data.api.Request. It prevents any thread from accessing this site while waiting.

Changed in version 11.10: Prioritize the API-reported lagtime over the Retry-After header when determining the wait time.

Parameters:

lagtime (float | None) – The time to wait for the next request which is the last maxlag time from api warning.

Return type:

None

modules: Counter[str]#
property next_multiplicity: float[source]#

Deprecated.

Factor to scale delay time based on upcoming request size.

Deprecated since version 10.3.0.

setDelays(delay=None, writedelay=None, absolute=False)[source]#

Deprecated; use set_delays instead.

Set the nominal delays in seconds.

Deprecated since version 10.3.0: Use set_delays() instead.

Parameters:

absolute (bool)

Return type:

None

set_delays(*, delay=None, writedelay=None, absolute=False)[source]#

Set the nominal delays in seconds.

Defaults to config values.

Added in version 10.3.0: Renamed from setDelays().

Parameters:

absolute (bool)

Return type:

None

static wait(seconds)[source]#

Wait for the specified number of seconds.

Announce the delay if it exceeds a preset limit.

Parameters:

seconds (int | float)

Return type:

None

waittime(write=False)[source]#

Return waiting time in seconds.

Changed in version 11.0: Use the latest request timestamp for read operation.

The result is for a query that would be made right now.

Parameters:

write (bool)

throttle.pid: bool | int = False#

Global process identifier.

When the first Throttle is instantiated, it will set this variable to a positive integer, which will apply to all throttle objects created by this process.