Enqueue and run background jobs via a federated queue, for wiki farms. More...

Inherits MediaWiki\JobQueue\JobQueue.

Collaboration diagram for MediaWiki\JobQueue\JobQueueFederated:

[legend]

Public Member Functions

getAllAbandonedJobs ()

Get an iterator to traverse over all abandoned jobs in this queue.

Stability: stable: to override

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

Since: 1.25

getAllAcquiredJobs ()

Get an iterator to traverse over all claimed jobs in this queue.Callers should be quick to iterator over it or few results will be returned due to jobs being acknowledged and deleted

Stability: stable: to override

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

Since: 1.26

getAllDelayedJobs ()

Get an iterator to traverse over all delayed jobs in this queue.Note: results may be stale if the queue is concurrently modified.

Stability: stable: to override

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

Since: 1.22

getAllQueuedJobs ()

Get an iterator to traverse over all available jobs in this queue.This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

getCoalesceLocationInternal ()

Do not use this function outside of JobQueue/JobQueueGroup.

Stability: stable: to override

Returns: string|null

Since: 1.22

Public Member Functions inherited from MediaWiki\JobQueue\JobQueue

ack (RunnableJob $job)

Acknowledge that a job was completed.

batchPush (array $jobs, $flags=0)

Push a batch of jobs into the queue.

deduplicateRootJob (IJobSpecification $job)

Register the "root job" of a given job into the queue for de-duplication.

delayedJobsEnabled ()

delete ()

Delete all unclaimed and delayed jobs from the queue.

flushCaches ()

Clear any process and persistent caches.

getAbandonedCount ()

Get the number of acquired jobs that can no longer be attempted.

getAcquiredCount ()

Get the number of acquired jobs (these are temporarily out of the queue).

getDelayedCount ()

Get the number of delayed jobs (these are temporarily out of the queue).

getDomain ()

getOrder ()

getReadOnlyReason ()

getSiblingQueueSizes (array $types)

Check the size of each of the given queues.

getSiblingQueuesWithJobs (array $types)

Check whether each of the given queues are empty.

getSize ()

Get the number of available (unacquired, non-delayed) jobs in the queue.

getType ()

isEmpty ()

Quickly check if the queue has no available (unacquired, non-delayed) jobs.

pop ()

Pop a job off of the queue.

push ( $jobs, $flags=0)

Push one or more jobs into the queue.

waitForBackups ()

Wait for any replica DBs or backup servers to catch up.

Protected Member Functions

__construct (array $params)

doAck (RunnableJob $job)

See also: JobQueue::ack()

Parameters

RunnableJob $job

doBatchPush (array $jobs, $flags)

See also: JobQueue::batchPush()

Parameters

IJobSpecification[]	$jobs
int	$flags

Exceptions

JobQueueError

doDeduplicateRootJob (IJobSpecification $job)

Stability: stable: to override

See also: JobQueue::deduplicateRootJob()

Parameters

IJobSpecification $job

Exceptions

JobQueueError

Returns: bool

doDelete ()

Stability: stable: to override

See also: JobQueue::delete()

Exceptions

JobQueueError

doFlushCaches ()

Stability: stable: to override

See also: JobQueue::flushCaches()

Returns: void

doGetAbandonedCount ()

Stability: stable: to override

See also: JobQueue::getAbandonedCount()

Returns: int

doGetAcquiredCount ()

See also: JobQueue::getAcquiredCount()

Returns: int

doGetDelayedCount ()

Stability: stable: to override

See also: JobQueue::getDelayedCount()

Returns: int

doGetSiblingQueueSizes (array $types)

Stability: stable: to override

See also: JobQueue::getSiblingQueuesSize()

Parameters

array $types List of queues types

Returns: array|null (list of queue types) or null if unsupported

doGetSiblingQueuesWithJobs (array $types)

Stability: stable: to override

See also: JobQueue::getSiblingQueuesWithJobs()

Parameters

array $types List of queues types

Returns: array|null (list of queue types) or null if unsupported

doGetSize ()

See also: JobQueue::getSize()

Returns: int

doIsEmpty ()

See also: JobQueue::isEmpty()

Returns: bool

doIsRootJobOldDuplicate (IJobSpecification $job)

Stability: stable: to override

See also: JobQueue::isRootJobOldDuplicate()

Parameters

IJobSpecification $job

Returns: bool

doPop ()

See also: JobQueue::pop()

Returns: RunnableJob|false

doWaitForBackups ()

Stability: stable: to override

See also: JobQueue::waitForBackups()

Returns: void

getCrossPartitionSum ( $type, $method)

logException (Exception $e)

optimalOrder ()

Get the default queue order to use if configuration does not specify one.

Returns: string One of (random, timestamp, fifo, undefined)

supportedOrders ()

Get the allowed queue orders for configuration validation.

Returns: array Subset of (random, timestamp, fifo, undefined)

supportsDelayedJobs ()

Find out if delayed jobs are supported for configuration validation.

Stability: stable: to override

Returns: bool Whether delayed jobs are supported

throwErrorIfAllPartitionsDown ( $down)

Throw an error if no partitions available.

tryJobInsertions (array $jobs, HashRing &$partitionRing, $flags)

Protected Member Functions inherited from MediaWiki\JobQueue\JobQueue

assertNotReadOnly ()

factoryJob ( $command, $params)

getRootJobCacheKey ( $signature, $type)

incrStats ( $event, $type, $delta=1)

Call StatsFactory::incrementBy() for the queue overall and for the queue type.

isRootJobOldDuplicate (IJobSpecification $job)

Check if the "root" job of a given job has been superseded by a newer one.

supportsTypeAgnostic ()

Subclasses should set this to true if they support type agnostic queues.

Protected Attributes
int	$maxPartitionsTry
	Maximum number of partitions to try.

JobQueue[]	$partitionQueues = []
	(partition name => JobQueue) reverse sorted by weight

HashRing	$partitionRing

Protected Attributes inherited from MediaWiki\JobQueue\JobQueue
int	$claimTTL
	Time to live in seconds.

string	$domain
	DB domain ID.

GlobalIdGenerator	$idGenerator

int	$maxTries
	Maximum number of times to try a job.

string	$order
	Job priority for pop()

string false	$readOnlyReason
	Read only rationale (or false if r/w)

StatsFactory	$stats

string	$type
	Job type.

bool	$typeAgnostic

WANObjectCache	$wanCache

const	QOS_ATOMIC = 1

const	ROOTJOB_TTL = 28 * 24 * 3600

Additional Inherited Members
Static Public Member Functions inherited from MediaWiki\JobQueue\JobQueue
static	factory (array $params)
	Get a job queue object of the specified type.

Detailed Description

Enqueue and run background jobs via a federated queue, for wiki farms.

This class allows for queues to be partitioned into smaller queues. A partition is defined by the configuration for a JobQueue instance. For example, one can set $wgJobTypeConf['refreshLinks'] to point to a JobQueueFederated instance, which itself would consist of three JobQueueRedis instances, each using their own redis server. This would allow for the jobs to be split (evenly or based on weights) across multiple servers if a single server becomes impractical or expensive. Different JobQueue classes can be mixed.

The basic queue configuration (e.g. "order", "claimTTL") of a federated queue is inherited by the partition queues. Additional configuration defines what section each wiki is in, what partition queues each section uses (and their weight), and the JobQueue configuration for each partition. Some sections might only need a single queue partition, like the sections for groups of small wikis.

If used for performance, then $wgMainCacheType should be set to memcached/redis. Note that "fifo" cannot be used for the ordering, since the data is distributed. One can still use "timestamp" instead, as in "roughly timestamp ordered". Also, queue classes used by this should ignore down servers (with TTL) to avoid slowness.

Since: 1.22

Definition at line 42 of file JobQueueFederated.php.

Constructor & Destructor Documentation

◆ __construct()

MediaWiki\JobQueue\JobQueueFederated::__construct ( array $params )

protected

Parameters

array

$params

Possible keys:

sectionsByWiki : A map of wiki IDs to section names. Wikis will default to using the section "default".
partitionsBySection : Map of section names to maps of (partition name => weight). A section called 'default' must be defined if not all wikis have explicitly defined sections.
configByPartition : Map of queue partition names to configuration arrays. These configuration arrays are passed to JobQueue::factory(). The options set here are overridden by those passed to this the federated queue itself (e.g. 'order' and 'claimTTL').
maxPartitionsTry : Maximum number of times to attempt job insertion using different partition queues. This improves availability during failure, at the cost of added latency and somewhat less reliable job de-duplication mechanisms.

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 67 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueue\$domain, and MediaWiki\JobQueue\JobQueue\factory().

Member Function Documentation

◆ doAck()

MediaWiki\JobQueue\JobQueueFederated::doAck ( RunnableJob $job )

protected

See also: JobQueue::ack()

Parameters

RunnableJob $job

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 304 of file JobQueueFederated.php.

References $job.

◆ doBatchPush()

MediaWiki\JobQueue\JobQueueFederated::doBatchPush	(	array	$jobs,
			$flags )

protected

See also: JobQueue::batchPush()

Parameters

IJobSpecification[]	$jobs
int	$flags

Exceptions

JobQueueError

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 177 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\$partitionRing, Wikimedia\HashRing\HashRing\getLiveLocationWeights(), and MediaWiki\JobQueue\JobQueueFederated\tryJobInsertions().

◆ doDeduplicateRootJob()

MediaWiki\JobQueue\JobQueueFederated::doDeduplicateRootJob ( IJobSpecification $job )

protected

Stability: stable: to override

See also: JobQueue::deduplicateRootJob()

Parameters

IJobSpecification $job

Exceptions

JobQueueError

Returns: bool

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 330 of file JobQueueFederated.php.

References $job.

◆ doDelete()

MediaWiki\JobQueue\JobQueueFederated::doDelete ( )

protected

Stability: stable: to override

See also: JobQueue::delete()

Exceptions

JobQueueError

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 346 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\logException(), and MediaWiki\JobQueue\JobQueueFederated\throwErrorIfAllPartitionsDown().

◆ doFlushCaches()

MediaWiki\JobQueue\JobQueueFederated::doFlushCaches ( )

protected

Stability: stable: to override

See also: JobQueue::flushCaches()

Returns: void

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 377 of file JobQueueFederated.php.

◆ doGetAbandonedCount()

MediaWiki\JobQueue\JobQueueFederated::doGetAbandonedCount ( )

protected

Stability: stable: to override

See also: JobQueue::getAbandonedCount()

Returns: int

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 151 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\getCrossPartitionSum().

◆ doGetAcquiredCount()

MediaWiki\JobQueue\JobQueueFederated::doGetAcquiredCount ( )

protected

See also: JobQueue::getAcquiredCount()

Returns: int

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 141 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\getCrossPartitionSum().

◆ doGetDelayedCount()

MediaWiki\JobQueue\JobQueueFederated::doGetDelayedCount ( )

protected

Stability: stable: to override

See also: JobQueue::getDelayedCount()

Returns: int

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 146 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\getCrossPartitionSum().

◆ doGetSiblingQueueSizes()

MediaWiki\JobQueue\JobQueueFederated::doGetSiblingQueueSizes ( array $types )

protected

Stability: stable: to override

See also: JobQueue::getSiblingQueuesSize()

Parameters

array $types List of queues types

Returns: array|null (list of queue types) or null if unsupported

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 466 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueue\$type, MediaWiki\JobQueue\JobQueueFederated\logException(), and MediaWiki\JobQueue\JobQueueFederated\throwErrorIfAllPartitionsDown().

◆ doGetSiblingQueuesWithJobs()

MediaWiki\JobQueue\JobQueueFederated::doGetSiblingQueuesWithJobs ( array $types )

protected

Stability: stable: to override

See also: JobQueue::getSiblingQueuesWithJobs()

Parameters

array $types List of queues types

Returns: array|null (list of queue types) or null if unsupported

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 439 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\logException(), and MediaWiki\JobQueue\JobQueueFederated\throwErrorIfAllPartitionsDown().

◆ doGetSize()

MediaWiki\JobQueue\JobQueueFederated::doGetSize ( )

protected

See also: JobQueue::getSize()

Returns: int

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 136 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\getCrossPartitionSum().

◆ doIsEmpty()

MediaWiki\JobQueue\JobQueueFederated::doIsEmpty ( )

protected

See also: JobQueue::isEmpty()

Returns: bool

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 119 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\logException(), and MediaWiki\JobQueue\JobQueueFederated\throwErrorIfAllPartitionsDown().

◆ doIsRootJobOldDuplicate()

MediaWiki\JobQueue\JobQueueFederated::doIsRootJobOldDuplicate ( IJobSpecification $job )

protected

Stability: stable: to override

See also: JobQueue::isRootJobOldDuplicate()

Parameters

IJobSpecification $job

Returns: bool

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 314 of file JobQueueFederated.php.

References $job.

◆ doPop()

MediaWiki\JobQueue\JobQueueFederated::doPop ( )

protected

See also: JobQueue::pop()

Returns: RunnableJob|false

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 271 of file JobQueueFederated.php.

References $job, MediaWiki\JobQueue\JobQueueFederated\logException(), and MediaWiki\JobQueue\JobQueueFederated\throwErrorIfAllPartitionsDown().

◆ doWaitForBackups()

MediaWiki\JobQueue\JobQueueFederated::doWaitForBackups ( )

protected

Stability: stable: to override

See also: JobQueue::waitForBackups()

Returns: void

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 362 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\logException(), and MediaWiki\JobQueue\JobQueueFederated\throwErrorIfAllPartitionsDown().

◆ getAllAbandonedJobs()

MediaWiki\JobQueue\JobQueueFederated::getAllAbandonedJobs ( )

Get an iterator to traverse over all abandoned jobs in this queue.

Stability: stable: to override

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

Since: 1.25

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 421 of file JobQueueFederated.php.

◆ getAllAcquiredJobs()

MediaWiki\JobQueue\JobQueueFederated::getAllAcquiredJobs ( )

Get an iterator to traverse over all claimed jobs in this queue.Callers should be quick to iterator over it or few results will be returned due to jobs being acknowledged and deleted

Stability: stable: to override

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

Since: 1.26

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 409 of file JobQueueFederated.php.

◆ getAllDelayedJobs()

MediaWiki\JobQueue\JobQueueFederated::getAllDelayedJobs ( )

Get an iterator to traverse over all delayed jobs in this queue.Note: results may be stale if the queue is concurrently modified.

Stability: stable: to override

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

Since: 1.22

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 397 of file JobQueueFederated.php.

◆ getAllQueuedJobs()

MediaWiki\JobQueue\JobQueueFederated::getAllQueuedJobs ( )

Get an iterator to traverse over all available jobs in this queue.This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.

Returns: \Iterator<RunnableJob>

Exceptions

JobQueueError

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 385 of file JobQueueFederated.php.

◆ getCoalesceLocationInternal()

MediaWiki\JobQueue\JobQueueFederated::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Stability: stable: to override

Returns: string|null

Since: 1.22

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 433 of file JobQueueFederated.php.

◆ getCrossPartitionSum()

MediaWiki\JobQueue\JobQueueFederated::getCrossPartitionSum	(		$type,
			$method )

protected

Parameters

string	$type
string	$method

Returns: int

Definition at line 160 of file JobQueueFederated.php.

References MediaWiki\JobQueue\JobQueueFederated\logException(), and MediaWiki\JobQueue\JobQueueFederated\throwErrorIfAllPartitionsDown().

Referenced by MediaWiki\JobQueue\JobQueueFederated\doGetAbandonedCount(), MediaWiki\JobQueue\JobQueueFederated\doGetAcquiredCount(), MediaWiki\JobQueue\JobQueueFederated\doGetDelayedCount(), and MediaWiki\JobQueue\JobQueueFederated\doGetSize().

◆ logException()

MediaWiki\JobQueue\JobQueueFederated::logException ( Exception $e )

protected

Definition at line 490 of file JobQueueFederated.php.

References wfDebugLog().

Referenced by MediaWiki\JobQueue\JobQueueFederated\doDelete(), MediaWiki\JobQueue\JobQueueFederated\doGetSiblingQueueSizes(), MediaWiki\JobQueue\JobQueueFederated\doGetSiblingQueuesWithJobs(), MediaWiki\JobQueue\JobQueueFederated\doIsEmpty(), MediaWiki\JobQueue\JobQueueFederated\doPop(), MediaWiki\JobQueue\JobQueueFederated\doWaitForBackups(), MediaWiki\JobQueue\JobQueueFederated\getCrossPartitionSum(), and MediaWiki\JobQueue\JobQueueFederated\tryJobInsertions().

◆ optimalOrder()

MediaWiki\JobQueue\JobQueueFederated::optimalOrder ( )

protected

Get the default queue order to use if configuration does not specify one.

Returns: string One of (random, timestamp, fifo, undefined)

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 103 of file JobQueueFederated.php.

◆ supportedOrders()

MediaWiki\JobQueue\JobQueueFederated::supportedOrders ( )

protected

Get the allowed queue orders for configuration validation.

Returns: array Subset of (random, timestamp, fifo, undefined)

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 97 of file JobQueueFederated.php.

◆ supportsDelayedJobs()

MediaWiki\JobQueue\JobQueueFederated::supportsDelayedJobs ( )

protected

Find out if delayed jobs are supported for configuration validation.

Stability: stable: to override

Returns: bool Whether delayed jobs are supported

Reimplemented from MediaWiki\JobQueue\JobQueue.

Definition at line 108 of file JobQueueFederated.php.

◆ throwErrorIfAllPartitionsDown()

MediaWiki\JobQueue\JobQueueFederated::throwErrorIfAllPartitionsDown ( $down )

protected

Throw an error if no partitions available.

Parameters

int $down The number of up partitions down

Returns: void

Exceptions

JobQueueError

Definition at line 501 of file JobQueueFederated.php.

Referenced by MediaWiki\JobQueue\JobQueueFederated\doDelete(), MediaWiki\JobQueue\JobQueueFederated\doGetSiblingQueueSizes(), MediaWiki\JobQueue\JobQueueFederated\doGetSiblingQueuesWithJobs(), MediaWiki\JobQueue\JobQueueFederated\doIsEmpty(), MediaWiki\JobQueue\JobQueueFederated\doPop(), MediaWiki\JobQueue\JobQueueFederated\doWaitForBackups(), and MediaWiki\JobQueue\JobQueueFederated\getCrossPartitionSum().

◆ tryJobInsertions()

MediaWiki\JobQueue\JobQueueFederated::tryJobInsertions	(	array	$jobs,
		HashRing &	$partitionRing,
			$flags )

protected

Parameters

array	$jobs
HashRing	&$partitionRing
int	$flags

Exceptions

JobQueueError

Returns: IJobSpecification[] List of Job object that could not be inserted

Definition at line 204 of file JobQueueFederated.php.

References $job, MediaWiki\JobQueue\JobQueueFederated\$partitionRing, Wikimedia\HashRing\HashRing\ejectFromLiveRing(), Wikimedia\HashRing\HashRing\getLiveLocation(), Wikimedia\HashRing\HashRing\getLiveLocationWeights(), and MediaWiki\JobQueue\JobQueueFederated\logException().

Referenced by MediaWiki\JobQueue\JobQueueFederated\doBatchPush().

Member Data Documentation

◆ $maxPartitionsTry

int MediaWiki\JobQueue\JobQueueFederated::$maxPartitionsTry

protected

Maximum number of partitions to try.

Definition at line 49 of file JobQueueFederated.php.

◆ $partitionQueues

JobQueue [] MediaWiki\JobQueue\JobQueueFederated::$partitionQueues = []

protected

(partition name => JobQueue) reverse sorted by weight

Definition at line 46 of file JobQueueFederated.php.

◆ $partitionRing

HashRing MediaWiki\JobQueue\JobQueueFederated::$partitionRing

protected

Definition at line 44 of file JobQueueFederated.php.

Referenced by MediaWiki\JobQueue\JobQueueFederated\doBatchPush(), and MediaWiki\JobQueue\JobQueueFederated\tryJobInsertions().

The documentation for this class was generated from the following file:

includes/JobQueue/JobQueueFederated.php

Public Member Functions

Protected Member Functions

Protected Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ doAck()

◆ doBatchPush()

◆ doDeduplicateRootJob()

◆ doDelete()

◆ doFlushCaches()

◆ doGetAbandonedCount()

◆ doGetAcquiredCount()

◆ doGetDelayedCount()

◆ doGetSiblingQueueSizes()

◆ doGetSiblingQueuesWithJobs()

◆ doGetSize()

◆ doIsEmpty()

◆ doIsRootJobOldDuplicate()

◆ doPop()

◆ doWaitForBackups()

◆ getAllAbandonedJobs()

◆ getAllAcquiredJobs()

◆ getAllDelayedJobs()

◆ getAllQueuedJobs()

◆ getCoalesceLocationInternal()

◆ getCrossPartitionSum()

◆ logException()

◆ optimalOrder()

◆ supportedOrders()

◆ supportsDelayedJobs()

◆ throwErrorIfAllPartitionsDown()

◆ tryJobInsertions()

Member Data Documentation

◆ $maxPartitionsTry

◆ $partitionQueues

◆ $partitionRing