MediaWiki  1.28.0
JobQueueFederated Class Reference

Class to handle enqueueing and running of background jobs for federated queues. More...

Inheritance diagram for JobQueueFederated:
Collaboration diagram for JobQueueFederated:

Public Member Functions

 getAllAbandonedJobs ()
 
 getAllAcquiredJobs ()
 
 getAllDelayedJobs ()
 
 getAllQueuedJobs ()
 
 getCoalesceLocationInternal ()
 
- Public Member Functions inherited from JobQueue
 ack (Job $job)
 Acknowledge that a job was completed. More...
 
 batchPush (array $jobs, $flags=0)
 Push a batch of jobs into the queue. More...
 
 deduplicateRootJob (IJobSpecification $job)
 Register the "root job" of a given job into the queue for de-duplication. More...
 
 delayedJobsEnabled ()
 
 delete ()
 Deleted all unclaimed and delayed jobs from the queue. More...
 
 flushCaches ()
 Clear any process and persistent caches. More...
 
 getAbandonedCount ()
 Get the number of acquired jobs that can no longer be attempted. More...
 
 getAcquiredCount ()
 Get the number of acquired jobs (these are temporarily out of the queue). More...
 
 getAllAbandonedJobs ()
 Get an iterator to traverse over all abandoned jobs in this queue. More...
 
 getAllAcquiredJobs ()
 Get an iterator to traverse over all claimed jobs in this queue. More...
 
 getAllDelayedJobs ()
 Get an iterator to traverse over all delayed jobs in this queue. More...
 
 getAllQueuedJobs ()
 Get an iterator to traverse over all available jobs in this queue. More...
 
 getCoalesceLocationInternal ()
 Do not use this function outside of JobQueue/JobQueueGroup. More...
 
 getDelayedCount ()
 Get the number of delayed jobs (these are temporarily out of the queue). More...
 
 getOrder ()
 
 getReadOnlyReason ()
 
 getSiblingQueueSizes (array $types)
 Check the size of each of the given queues. More...
 
 getSiblingQueuesWithJobs (array $types)
 Check whether each of the given queues are empty. More...
 
 getSize ()
 Get the number of available (unacquired, non-delayed) jobs in the queue. More...
 
 getType ()
 
 getWiki ()
 
 isEmpty ()
 Quickly check if the queue has no available (unacquired, non-delayed) jobs. More...
 
 pop ()
 Pop a job off of the queue. More...
 
 push ($jobs, $flags=0)
 Push one or more jobs into the queue. More...
 
 waitForBackups ()
 Wait for any replica DBs or backup servers to catch up. More...
 

Protected Member Functions

 __construct (array $params)
 
 doAck (Job $job)
 
 doBatchPush (array $jobs, $flags)
 
 doDeduplicateRootJob (IJobSpecification $job)
 
 doDelete ()
 
 doFlushCaches ()
 
 doGetAbandonedCount ()
 
 doGetAcquiredCount ()
 
 doGetDelayedCount ()
 
 doGetSiblingQueueSizes (array $types)
 
 doGetSiblingQueuesWithJobs (array $types)
 
 doGetSize ()
 
 doIsEmpty ()
 
 doIsRootJobOldDuplicate (Job $job)
 
 doPop ()
 
 doWaitForBackups ()
 
 getCrossPartitionSum ($type, $method)
 
 logException (Exception $e)
 
 optimalOrder ()
 
 supportedOrders ()
 
 supportsDelayedJobs ()
 
 throwErrorIfAllPartitionsDown ($down)
 Throw an error if no partitions available. More...
 
 tryJobInsertions (array $jobs, HashRing &$partitionRing, $flags)
 
- Protected Member Functions inherited from JobQueue
 __construct (array $params)
 
 assertNotReadOnly ()
 
 doAck (Job $job)
 
 doBatchPush (array $jobs, $flags)
 
 doDeduplicateRootJob (IJobSpecification $job)
 
 doDelete ()
 
 doFlushCaches ()
 
 doGetAbandonedCount ()
 
 doGetAcquiredCount ()
 
 doGetDelayedCount ()
 
 doGetSiblingQueueSizes (array $types)
 
 doGetSiblingQueuesWithJobs (array $types)
 
 doGetSize ()
 
 doIsEmpty ()
 
 doIsRootJobOldDuplicate (Job $job)
 
 doPop ()
 
 doWaitForBackups ()
 
 getRootJobCacheKey ($signature)
 
 isRootJobOldDuplicate (Job $job)
 Check if the "root" job of a given job has been superseded by a newer one. More...
 
 optimalOrder ()
 Get the default queue order to use if configuration does not specify one. More...
 
 supportedOrders ()
 Get the allowed queue orders for configuration validation. More...
 
 supportsDelayedJobs ()
 Find out if delayed jobs are supported for configuration validation. More...
 

Protected Attributes

int $maxPartitionsTry
 Maximum number of partitions to try. More...
 
JobQueue[] $partitionQueues = []
 (partition name => JobQueue) reverse sorted by weight More...
 
HashRing $partitionRing
 
- Protected Attributes inherited from JobQueue
JobQueueAggregator $aggr
 
int $claimTTL
 Time to live in seconds. More...
 
BagOStuff $dupCache
 
int $maxTries
 Maximum number of times to try a job. More...
 
string $order
 Job priority for pop() More...
 
string bool $readOnlyReason
 Read only rationale (or false if r/w) More...
 
string $type
 Job type. More...
 
string $wiki
 Wiki ID. More...
 

Additional Inherited Members

- Static Public Member Functions inherited from JobQueue
static factory (array $params)
 Get a job queue object of the specified type. More...
 
static incrStats ($key, $type, $delta=1)
 Call wfIncrStats() for the queue overall and for the queue type. More...
 
- Public Attributes inherited from JobQueue
const QOS_ATOMIC = 1
 
const ROOTJOB_TTL = 2419200
 

Detailed Description

Class to handle enqueueing and running of background jobs for federated queues.

This class allows for queues to be partitioned into smaller queues. A partition is defined by the configuration for a JobQueue instance. For example, one can set $wgJobTypeConf['refreshLinks'] to point to a JobQueueFederated instance, which itself would consist of three JobQueueRedis instances, each using their own redis server. This would allow for the jobs to be split (evenly or based on weights) accross multiple servers if a single server becomes impractical or expensive. Different JobQueue classes can be mixed.

The basic queue configuration (e.g. "order", "claimTTL") of a federated queue is inherited by the partition queues. Additional configuration defines what section each wiki is in, what partition queues each section uses (and their weight), and the JobQueue configuration for each partition. Some sections might only need a single queue partition, like the sections for groups of small wikis.

If used for performance, then $wgMainCacheType should be set to memcached/redis. Note that "fifo" cannot be used for the ordering, since the data is distributed. One can still use "timestamp" instead, as in "roughly timestamp ordered". Also, queue classes used by this should ignore down servers (with TTL) to avoid slowness.

Since
1.22

Definition at line 49 of file JobQueueFederated.php.

Constructor & Destructor Documentation

JobQueueFederated::__construct ( array  $params)
protected
Parameters
array$paramsPossible keys:
  • sectionsByWiki : A map of wiki IDs to section names. Wikis will default to using the section "default".
  • partitionsBySection : Map of section names to maps of (partition name => weight). A section called 'default' must be defined if not all wikis have explicitly defined sections.
  • configByPartition : Map of queue partition names to configuration arrays. These configuration arrays are passed to JobQueue::factory(). The options set here are overridden by those passed to this the federated queue itself (e.g. 'order' and 'claimTTL').
  • maxPartitionsTry : Maximum number of times to attempt job insertion using different partition queues. This improves availability during failure, at the cost of added latency and somewhat less reliable job de-duplication mechanisms.
Exceptions
MWException

Definition at line 75 of file JobQueueFederated.php.

References $params, $section, JobQueue\$wiki, as, JobQueue\factory(), and wiki.

Member Function Documentation

JobQueueFederated::doAck ( Job  $job)
protected

Definition at line 309 of file JobQueueFederated.php.

JobQueueFederated::doBatchPush ( array  $jobs,
  $flags 
)
protected
JobQueueFederated::doDeduplicateRootJob ( IJobSpecification  $job)
protected

Definition at line 332 of file JobQueueFederated.php.

References $e, and IJobSpecification\getRootJobParams().

JobQueueFederated::doDelete ( )
protected

Definition at line 347 of file JobQueueFederated.php.

References $e, $queue, as, logException(), and throwErrorIfAllPartitionsDown().

JobQueueFederated::doFlushCaches ( )
protected

Definition at line 376 of file JobQueueFederated.php.

References $queue, and as.

JobQueueFederated::doGetAbandonedCount ( )
protected

Definition at line 157 of file JobQueueFederated.php.

References getCrossPartitionSum().

JobQueueFederated::doGetAcquiredCount ( )
protected

Definition at line 149 of file JobQueueFederated.php.

References getCrossPartitionSum().

JobQueueFederated::doGetDelayedCount ( )
protected

Definition at line 153 of file JobQueueFederated.php.

References getCrossPartitionSum().

JobQueueFederated::doGetSiblingQueueSizes ( array  $types)
protected
JobQueueFederated::doGetSiblingQueuesWithJobs ( array  $types)
protected

Definition at line 432 of file JobQueueFederated.php.

References $e, $queue, as, logException(), and throwErrorIfAllPartitionsDown().

JobQueueFederated::doGetSize ( )
protected

Definition at line 145 of file JobQueueFederated.php.

References getCrossPartitionSum().

JobQueueFederated::doIsEmpty ( )
protected

Definition at line 129 of file JobQueueFederated.php.

References $e, $queue, as, logException(), and throwErrorIfAllPartitionsDown().

JobQueueFederated::doIsRootJobOldDuplicate ( Job  $job)
protected

Definition at line 317 of file JobQueueFederated.php.

References $e, and Job\getRootJobParams().

JobQueueFederated::doPop ( )
protected
JobQueueFederated::doWaitForBackups ( )
protected

Definition at line 362 of file JobQueueFederated.php.

References $e, $queue, as, logException(), and throwErrorIfAllPartitionsDown().

JobQueueFederated::getAllAbandonedJobs ( )

Definition at line 416 of file JobQueueFederated.php.

References $queue, and as.

JobQueueFederated::getAllAcquiredJobs ( )

Definition at line 405 of file JobQueueFederated.php.

References $queue, and as.

JobQueueFederated::getAllDelayedJobs ( )

Definition at line 394 of file JobQueueFederated.php.

References $queue, and as.

JobQueueFederated::getAllQueuedJobs ( )

Definition at line 383 of file JobQueueFederated.php.

References $queue, and as.

JobQueueFederated::getCoalesceLocationInternal ( )

Definition at line 427 of file JobQueueFederated.php.

References serialize().

JobQueueFederated::getCrossPartitionSum (   $type,
  $method 
)
protected
Parameters
string$type
string$method
Returns
int

Definition at line 166 of file JobQueueFederated.php.

References $count, $e, $queue, as, logException(), and throwErrorIfAllPartitionsDown().

Referenced by doGetAbandonedCount(), doGetAcquiredCount(), doGetDelayedCount(), and doGetSize().

JobQueueFederated::logException ( Exception  $e)
protected
JobQueueFederated::optimalOrder ( )
protected

Definition at line 115 of file JobQueueFederated.php.

JobQueueFederated::supportedOrders ( )
protected

Definition at line 110 of file JobQueueFederated.php.

JobQueueFederated::supportsDelayedJobs ( )
protected

Definition at line 119 of file JobQueueFederated.php.

References $queue, and as.

JobQueueFederated::throwErrorIfAllPartitionsDown (   $down)
protected

Throw an error if no partitions available.

Parameters
int$downThe number of up partitions down
Returns
void
Exceptions
JobQueueError

Definition at line 493 of file JobQueueFederated.php.

Referenced by doDelete(), doGetSiblingQueueSizes(), doGetSiblingQueuesWithJobs(), doIsEmpty(), doPop(), doWaitForBackups(), and getCrossPartitionSum().

JobQueueFederated::tryJobInsertions ( array  $jobs,
HashRing $partitionRing,
  $flags 
)
protected
Parameters
array$jobs
HashRing$partitionRing
int$flags
Exceptions
JobQueueError
Returns
array List of Job object that could not be inserted

Definition at line 211 of file JobQueueFederated.php.

References $e, $flags, $job, $queue, as, HashRing\ejectFromLiveRing(), HashRing\getLiveLocation(), HashRing\getLiveLocationWeights(), logException(), ArrayUtils\pickRandom(), and serialize().

Referenced by doBatchPush().

Member Data Documentation

int JobQueueFederated::$maxPartitionsTry
protected

Maximum number of partitions to try.

Definition at line 56 of file JobQueueFederated.php.

JobQueue [] JobQueueFederated::$partitionQueues = []
protected

(partition name => JobQueue) reverse sorted by weight

Definition at line 53 of file JobQueueFederated.php.

HashRing JobQueueFederated::$partitionRing
protected

Definition at line 51 of file JobQueueFederated.php.

Referenced by doBatchPush().


The documentation for this class was generated from the following file: