MediaWiki  1.23.15
JobQueueFederated Class Reference

Class to handle enqueueing and running of background jobs for federated queues. More...

Inheritance diagram for JobQueueFederated:
Collaboration diagram for JobQueueFederated:

Public Member Functions

 getAllDelayedJobs ()
 Get an iterator to traverse over all delayed jobs in this queue. More...
 
 getAllQueuedJobs ()
 Get an iterator to traverse over all available jobs in this queue. More...
 
 getCoalesceLocationInternal ()
 Do not use this function outside of JobQueue/JobQueueGroup. More...
 
 setTestingPrefix ( $key)
 Namespace the queue with a key to isolate it for testing. More...
 
- Public Member Functions inherited from JobQueue
 ack (Job $job)
 Acknowledge that a job was completed. More...
 
 batchPush (array $jobs, $flags=0)
 Push a batch of jobs into the queue. More...
 
 deduplicateRootJob (Job $job)
 Register the "root job" of a given job into the queue for de-duplication. More...
 
 delayedJobsEnabled ()
 
 delete ()
 Deleted all unclaimed and delayed jobs from the queue. More...
 
 flushCaches ()
 Clear any process and persistent caches. More...
 
 getAbandonedCount ()
 Get the number of acquired jobs that can no longer be attempted. More...
 
 getAcquiredCount ()
 Get the number of acquired jobs (these are temporarily out of the queue). More...
 
 getDelayedCount ()
 Get the number of delayed jobs (these are temporarily out of the queue). More...
 
 getOrder ()
 
 getPeriodicTasks ()
 Return a map of task names to task definition maps. More...
 
 getSiblingQueueSizes (array $types)
 Check the size of each of the given queues. More...
 
 getSiblingQueuesWithJobs (array $types)
 Check whether each of the given queues are empty. More...
 
 getSize ()
 Get the number of available (unacquired, non-delayed) jobs in the queue. More...
 
 getType ()
 
 getWiki ()
 
 isEmpty ()
 Quickly check if the queue has no available (unacquired, non-delayed) jobs. More...
 
 pop ()
 Pop a job off of the queue. More...
 
 push ( $jobs, $flags=0)
 Push one or more jobs into the queue. More...
 
 waitForBackups ()
 Wait for any slaves or backup servers to catch up. More...
 

Public Attributes

const CACHE_TTL_LONG = 300
 
const CACHE_TTL_SHORT = 30
 
- Public Attributes inherited from JobQueue
const QOS_ATOMIC = 1
 
const ROOTJOB_TTL = 2419200
 

Protected Member Functions

 __construct (array $params)
 @params include: More...
 
 doAck (Job $job)
 
 doBatchPush (array $jobs, $flags)
 
 doDeduplicateRootJob (Job $job)
 
 doDelete ()
 
 doFlushCaches ()
 
 doGetAbandonedCount ()
 
 doGetAcquiredCount ()
 
 doGetDelayedCount ()
 
 doGetPeriodicTasks ()
 
 doGetSiblingQueueSizes (array $types)
 
 doGetSiblingQueuesWithJobs (array $types)
 
 doGetSize ()
 
 doIsEmpty ()
 
 doIsRootJobOldDuplicate (Job $job)
 
 doPop ()
 
 doWaitForBackups ()
 
 getCrossPartitionSum ( $type, $method)
 
 optimalOrder ()
 Get the default queue order to use if configuration does not specify one. More...
 
 supportedOrders ()
 Get the allowed queue orders for configuration validation. More...
 
 supportsDelayedJobs ()
 Find out if delayed jobs are supported for configuration validation. More...
 
 throwErrorIfAllPartitionsDown ( $down)
 Throw an error if no partitions available. More...
 
 tryJobInsertions (array $jobs, HashRing &$partitionRing, $flags)
 
- Protected Member Functions inherited from JobQueue
 getRootJobCacheKey ( $signature)
 
 isRootJobOldDuplicate (Job $job)
 Check if the "root" job of a given job has been superseded by a newer one. More...
 

Protected Attributes

BagOStuff $cache
 
int $maxPartitionsTry
 Maximum number of partitions to try *. More...
 
array $partitionMap = array()
 (partition name => weight) reverse sorted by weight * More...
 
HashRing $partitionPushRing
 
array $partitionQueues = array()
 (partition name => JobQueue) reverse sorted by weight * More...
 
- Protected Attributes inherited from JobQueue
bool $checkDelay
 Allow delayed jobs *. More...
 
int $claimTTL
 Time to live in seconds *. More...
 
BagOStuff $dupCache
 
int $maxTries
 Maximum number of times to try a job *. More...
 
string $order
 Job priority for pop() *. More...
 
string $type
 Job type *. More...
 
string $wiki
 Wiki ID *. More...
 

Private Member Functions

 getCacheKey ( $property)
 

Additional Inherited Members

- Static Public Member Functions inherited from JobQueue
static factory (array $params)
 Get a job queue object of the specified type. More...
 
static incrStats ( $key, $type, $delta=1, $wiki=null)
 Call wfIncrStats() for the queue overall and for the queue type. More...
 

Detailed Description

Class to handle enqueueing and running of background jobs for federated queues.

This class allows for queues to be partitioned into smaller queues. A partition is defined by the configuration for a JobQueue instance. For example, one can set $wgJobTypeConf['refreshLinks'] to point to a JobQueueFederated instance, which itself would consist of three JobQueueRedis instances, each using their own redis server. This would allow for the jobs to be split (evenly or based on weights) accross multiple servers if a single server becomes impractical or expensive. Different JobQueue classes can be mixed.

The basic queue configuration (e.g. "order", "claimTTL") of a federated queue is inherited by the partition queues. Additional configuration defines what section each wiki is in, what partition queues each section uses (and their weight), and the JobQueue configuration for each partition. Some sections might only need a single queue partition, like the sections for groups of small wikis.

If used for performance, then $wgMainCacheType should be set to memcached/redis. Note that "fifo" cannot be used for the ordering, since the data is distributed. One can still use "timestamp" instead, as in "roughly timestamp ordered". Also, queue classes used by this should ignore down servers (with TTL) to avoid slowness.

Since
1.22

Definition at line 49 of file JobQueueFederated.php.

Constructor & Destructor Documentation

◆ __construct()

JobQueueFederated::__construct ( array  $params)
protected

@params include:

  • sectionsByWiki : A map of wiki IDs to section names. Wikis will default to using the section "default".
  • partitionsBySection : Map of section names to maps of (partition name => weight). A section called 'default' must be defined if not all wikis have explicitly defined sections.
  • configByPartition : Map of queue partition names to configuration arrays. These configuration arrays are passed to JobQueue::factory(). The options set here are overriden by those passed to this the federated queue itself (e.g. 'order' and 'claimTTL').
  • partitionsNoPush : List of partition names that can handle pop() but not push(). This can be used to migrate away from a certain partition.
  • maxPartitionsTry : Maximum number of times to attempt job insertion using different partition queues. This improves availability during failure, at the cost of added latency and somewhat less reliable job de-duplication mechanisms.
    Parameters
    array$params
    Exceptions
    MWException

Reimplemented from JobQueue.

Definition at line 83 of file JobQueueFederated.php.

References $params, $partitionMap, $section, JobQueue\$wiki, array(), as, cache, JobQueue\factory(), wfGetMainCache(), and wiki.

Member Function Documentation

◆ doAck()

JobQueueFederated::doAck ( Job  $job)
protected
See also
JobQueue::ack()
Parameters
Job$job
Returns
bool

Reimplemented from JobQueue.

Definition at line 346 of file JobQueueFederated.php.

References $job.

◆ doBatchPush()

JobQueueFederated::doBatchPush ( array  $jobs,
  $flags 
)
protected
See also
JobQueue::batchPush()
Parameters
array$jobs
$flags
Returns
bool

Reimplemented from JobQueue.

Definition at line 209 of file JobQueueFederated.php.

References $flags, $partitionPushRing, and tryJobInsertions().

◆ doDeduplicateRootJob()

JobQueueFederated::doDeduplicateRootJob ( Job  $job)
protected
See also
JobQueue::deduplicateRootJob()
Parameters
Job$job
Exceptions
MWException
Returns
bool

Reimplemented from JobQueue.

Definition at line 368 of file JobQueueFederated.php.

References $e, $job, and $params.

◆ doDelete()

JobQueueFederated::doDelete ( )
protected
See also
JobQueue::delete()
Exceptions
MWException
Returns
bool Success

Reimplemented from JobQueue.

Definition at line 382 of file JobQueueFederated.php.

References $e, $failed, as, MWExceptionHandler\logException(), and throwErrorIfAllPartitionsDown().

◆ doFlushCaches()

JobQueueFederated::doFlushCaches ( )
protected
See also
JobQueue::flushCaches()
Returns
void

Reimplemented from JobQueue.

Definition at line 423 of file JobQueueFederated.php.

References JobQueue\$type, array(), as, cache, and getCacheKey().

◆ doGetAbandonedCount()

JobQueueFederated::doGetAbandonedCount ( )
protected
See also
JobQueue::getAbandonedCount()
Returns
int

Reimplemented from JobQueue.

Definition at line 176 of file JobQueueFederated.php.

References getCrossPartitionSum().

◆ doGetAcquiredCount()

JobQueueFederated::doGetAcquiredCount ( )
protected
See also
JobQueue::getAcquiredCount()
Returns
int

Reimplemented from JobQueue.

Definition at line 168 of file JobQueueFederated.php.

References getCrossPartitionSum().

◆ doGetDelayedCount()

JobQueueFederated::doGetDelayedCount ( )
protected
See also
JobQueue::getDelayedCount()
Returns
int

Reimplemented from JobQueue.

Definition at line 172 of file JobQueueFederated.php.

References getCrossPartitionSum().

◆ doGetPeriodicTasks()

JobQueueFederated::doGetPeriodicTasks ( )
protected
See also
JobQueue::getPeriodicTasks()
Returns
array

Reimplemented from JobQueue.

Definition at line 411 of file JobQueueFederated.php.

References array(), and as.

◆ doGetSiblingQueueSizes()

JobQueueFederated::doGetSiblingQueueSizes ( array  $types)
protected
See also
JobQueue::getSiblingQueuesSize()
Parameters
array$typesList of queues types
Returns
array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 495 of file JobQueueFederated.php.

References $e, $failed, $size, JobQueue\$type, array(), as, MWExceptionHandler\logException(), and throwErrorIfAllPartitionsDown().

◆ doGetSiblingQueuesWithJobs()

JobQueueFederated::doGetSiblingQueuesWithJobs ( array  $types)
protected
See also
JobQueue::getSiblingQueuesWithJobs()
Parameters
array$typesList of queues types
Returns
array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 469 of file JobQueueFederated.php.

References $e, $failed, array(), as, MWExceptionHandler\logException(), and throwErrorIfAllPartitionsDown().

◆ doGetSize()

JobQueueFederated::doGetSize ( )
protected
See also
JobQueue::getSize()
Returns
int

Reimplemented from JobQueue.

Definition at line 164 of file JobQueueFederated.php.

References getCrossPartitionSum().

◆ doIsEmpty()

JobQueueFederated::doIsEmpty ( )
protected
See also
JobQueue::isEmpty()
Returns
bool

Reimplemented from JobQueue.

Definition at line 138 of file JobQueueFederated.php.

References $e, $failed, as, cache, getCacheKey(), MWExceptionHandler\logException(), and throwErrorIfAllPartitionsDown().

◆ doIsRootJobOldDuplicate()

JobQueueFederated::doIsRootJobOldDuplicate ( Job  $job)
protected
See also
JobQueue::isRootJobOldDuplicate()
Parameters
Job$job
Returns
bool

Reimplemented from JobQueue.

Definition at line 354 of file JobQueueFederated.php.

References $e, $job, and $params.

◆ doPop()

JobQueueFederated::doPop ( )
protected

◆ doWaitForBackups()

JobQueueFederated::doWaitForBackups ( )
protected
See also
JobQueue::waitForBackups()
Returns
void

Reimplemented from JobQueue.

Definition at line 397 of file JobQueueFederated.php.

References $e, $failed, as, MWExceptionHandler\logException(), and throwErrorIfAllPartitionsDown().

◆ getAllDelayedJobs()

JobQueueFederated::getAllDelayedJobs ( )

Get an iterator to traverse over all delayed jobs in this queue.

Note: results may be stale if the queue is concurrently modified.

Returns
Iterator
Exceptions
JobQueueError
Since
1.22

Reimplemented from JobQueue.

Definition at line 453 of file JobQueueFederated.php.

References as.

◆ getAllQueuedJobs()

JobQueueFederated::getAllQueuedJobs ( )

Get an iterator to traverse over all available jobs in this queue.

This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.

Returns
Iterator
Exceptions
JobQueueError

Reimplemented from JobQueue.

Definition at line 442 of file JobQueueFederated.php.

References as.

◆ getCacheKey()

JobQueueFederated::getCacheKey (   $property)
private
Parameters
$property
Returns
string

Definition at line 543 of file JobQueueFederated.php.

References $property, list, type, wfForeignMemcKey(), wfSplitWikiID(), and wiki.

Referenced by doFlushCaches(), doIsEmpty(), doPop(), getCrossPartitionSum(), and tryJobInsertions().

◆ getCoalesceLocationInternal()

JobQueueFederated::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Returns
string
Since
1.22

Reimplemented from JobQueue.

Definition at line 464 of file JobQueueFederated.php.

◆ getCrossPartitionSum()

JobQueueFederated::getCrossPartitionSum (   $type,
  $method 
)
protected

◆ optimalOrder()

JobQueueFederated::optimalOrder ( )
protected

Get the default queue order to use if configuration does not specify one.

Returns
string One of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 130 of file JobQueueFederated.php.

◆ setTestingPrefix()

JobQueueFederated::setTestingPrefix (   $key)

Namespace the queue with a key to isolate it for testing.

Parameters
string$key
Returns
void
Exceptions
MWException

Reimplemented from JobQueue.

Definition at line 532 of file JobQueueFederated.php.

References as.

◆ supportedOrders()

JobQueueFederated::supportedOrders ( )
protected

Get the allowed queue orders for configuration validation.

Returns
array Subset of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 125 of file JobQueueFederated.php.

References array().

◆ supportsDelayedJobs()

JobQueueFederated::supportsDelayedJobs ( )
protected

Find out if delayed jobs are supported for configuration validation.

Returns
bool Whether delayed jobs are supported

Reimplemented from JobQueue.

Definition at line 134 of file JobQueueFederated.php.

◆ throwErrorIfAllPartitionsDown()

JobQueueFederated::throwErrorIfAllPartitionsDown (   $down)
protected

Throw an error if no partitions available.

Parameters
int$downThe number of up partitions down
Returns
void
Exceptions
JobQueueError

Definition at line 526 of file JobQueueFederated.php.

Referenced by doDelete(), doGetSiblingQueueSizes(), doGetSiblingQueuesWithJobs(), doIsEmpty(), doPop(), doWaitForBackups(), and getCrossPartitionSum().

◆ tryJobInsertions()

JobQueueFederated::tryJobInsertions ( array  $jobs,
HashRing $partitionRing,
  $flags 
)
protected
Parameters
array$jobs
HashRing$partitionRing
int$flags
Exceptions
JobQueueError
Returns
array List of Job object that could not be inserted

Definition at line 233 of file JobQueueFederated.php.

References $e, $flags, $job, $ok, array(), as, cache, JobQueueDB\CACHE_TTL_LONG, getCacheKey(), HashRing\getLocation(), HashRing\getLocationWeights(), MWExceptionHandler\logException(), HashRing\newWithoutLocation(), and ArrayUtils\pickRandom().

Referenced by doBatchPush().

Member Data Documentation

◆ $cache

BagOStuff JobQueueFederated::$cache
protected

Definition at line 56 of file JobQueueFederated.php.

◆ $maxPartitionsTry

int JobQueueFederated::$maxPartitionsTry
protected

Maximum number of partitions to try *.

Definition at line 58 of file JobQueueFederated.php.

◆ $partitionMap

array JobQueueFederated::$partitionMap = array()
protected

(partition name => weight) reverse sorted by weight *

Definition at line 50 of file JobQueueFederated.php.

Referenced by __construct(), and doPop().

◆ $partitionPushRing

HashRing JobQueueFederated::$partitionPushRing
protected

Definition at line 54 of file JobQueueFederated.php.

Referenced by doBatchPush().

◆ $partitionQueues

array JobQueueFederated::$partitionQueues = array()
protected

(partition name => JobQueue) reverse sorted by weight *

Definition at line 52 of file JobQueueFederated.php.

◆ CACHE_TTL_LONG

const JobQueueFederated::CACHE_TTL_LONG = 300

Definition at line 61 of file JobQueueFederated.php.

◆ CACHE_TTL_SHORT

const JobQueueFederated::CACHE_TTL_SHORT = 30

Definition at line 60 of file JobQueueFederated.php.


The documentation for this class was generated from the following file: