MediaWiki  master
JobQueue Class Reference

Class to handle enqueueing and running of background jobs. More...

Inheritance diagram for JobQueue:
Collaboration diagram for JobQueue:

Public Member Functions

 ack (RunnableJob $job)
 Acknowledge that a job was completed. More...
 
 batchPush (array $jobs, $flags=0)
 Push a batch of jobs into the queue. More...
 
 deduplicateRootJob (IJobSpecification $job)
 Register the "root job" of a given job into the queue for de-duplication. More...
 
 delayedJobsEnabled ()
 
 delete ()
 Deleted all unclaimed and delayed jobs from the queue. More...
 
 flushCaches ()
 Clear any process and persistent caches. More...
 
 getAbandonedCount ()
 Get the number of acquired jobs that can no longer be attempted. More...
 
 getAcquiredCount ()
 Get the number of acquired jobs (these are temporarily out of the queue). More...
 
 getAllAbandonedJobs ()
 Get an iterator to traverse over all abandoned jobs in this queue. More...
 
 getAllAcquiredJobs ()
 Get an iterator to traverse over all claimed jobs in this queue. More...
 
 getAllDelayedJobs ()
 Get an iterator to traverse over all delayed jobs in this queue. More...
 
 getAllQueuedJobs ()
 Get an iterator to traverse over all available jobs in this queue. More...
 
 getCoalesceLocationInternal ()
 Do not use this function outside of JobQueue/JobQueueGroup. More...
 
 getDelayedCount ()
 Get the number of delayed jobs (these are temporarily out of the queue). More...
 
 getDomain ()
 
 getOrder ()
 
 getReadOnlyReason ()
 
 getSiblingQueueSizes (array $types)
 Check the size of each of the given queues. More...
 
 getSiblingQueuesWithJobs (array $types)
 Check whether each of the given queues are empty. More...
 
 getSize ()
 Get the number of available (unacquired, non-delayed) jobs in the queue. More...
 
 getType ()
 
 getWiki ()
 
 isEmpty ()
 Quickly check if the queue has no available (unacquired, non-delayed) jobs. More...
 
 pop ()
 Pop a job off of the queue. More...
 
 push ( $jobs, $flags=0)
 Push one or more jobs into the queue. More...
 
 waitForBackups ()
 Wait for any replica DBs or backup servers to catch up. More...
 

Static Public Member Functions

static factory (array $params)
 Get a job queue object of the specified type. More...
 

Public Attributes

const QOS_ATOMIC = 1
 
const ROOTJOB_TTL = 2419200
 

Protected Member Functions

 __construct (array $params)
 
 assertNotReadOnly ()
 
 doAck (RunnableJob $job)
 
 doBatchPush (array $jobs, $flags)
 
 doDeduplicateRootJob (IJobSpecification $job)
 
 doDelete ()
 
 doFlushCaches ()
 
 doGetAbandonedCount ()
 
 doGetAcquiredCount ()
 
 doGetDelayedCount ()
 
 doGetSiblingQueueSizes (array $types)
 
 doGetSiblingQueuesWithJobs (array $types)
 
 doGetSize ()
 
 doIsEmpty ()
 
 doIsRootJobOldDuplicate (IJobSpecification $job)
 
 doPop ()
 
 doWaitForBackups ()
 
 factoryJob ( $command, $params)
 
 getRootJobCacheKey ( $signature)
 
 incrStats ( $key, $type, $delta=1)
 Call wfIncrStats() for the queue overall and for the queue type. More...
 
 isRootJobOldDuplicate (IJobSpecification $job)
 Check if the "root" job of a given job has been superseded by a newer one. More...
 
 optimalOrder ()
 Get the default queue order to use if configuration does not specify one. More...
 
 supportedOrders ()
 Get the allowed queue orders for configuration validation. More...
 
 supportsDelayedJobs ()
 Find out if delayed jobs are supported for configuration validation. More...
 

Protected Attributes

int $claimTTL
 Time to live in seconds. More...
 
string $domain
 DB domain ID. More...
 
BagOStuff $dupCache
 
int $maxTries
 Maximum number of times to try a job. More...
 
string $order
 Job priority for pop() More...
 
string bool $readOnlyReason
 Read only rationale (or false if r/w) More...
 
StatsdDataFactoryInterface $stats
 
string $type
 Job type. More...
 

Detailed Description

Class to handle enqueueing and running of background jobs.

Since
1.21

Definition at line 31 of file JobQueue.php.

Constructor & Destructor Documentation

◆ __construct()

JobQueue::__construct ( array  $params)
protected
Parameters
array$params
Exceptions
JobQueueError

Definition at line 58 of file JobQueue.php.

References optimalOrder(), supportedOrders(), and type.

Member Function Documentation

◆ ack()

JobQueue::ack ( RunnableJob  $job)
final

Acknowledge that a job was completed.

This does nothing for certain queue classes or if "claimTTL" is not set. Outside callers should use JobQueueGroup::ack() instead of this function.

Parameters
RunnableJob$job
Returns
void
Exceptions
JobQueueError

Definition at line 400 of file JobQueue.php.

References $job, $type, assertNotReadOnly(), doAck(), and IJobSpecification\getType().

◆ assertNotReadOnly()

JobQueue::assertNotReadOnly ( )
protected
Exceptions
JobQueueReadOnlyError

Definition at line 702 of file JobQueue.php.

Referenced by ack(), batchPush(), deduplicateRootJob(), delete(), and pop().

◆ batchPush()

JobQueue::batchPush ( array  $jobs,
  $flags = 0 
)
final

Push a batch of jobs into the queue.

This does not require $wgJobClasses to be set for the given job type. Outside callers should use JobQueueGroup::push() instead of this function.

Parameters
IJobSpecification[]$jobs
int$flagsBitfield (supports JobQueue::QOS_ATOMIC)
Returns
void
Exceptions
JobQueueError

Definition at line 325 of file JobQueue.php.

References $job, $type, array(), as, assertNotReadOnly(), deduplicateRootJob(), doBatchPush(), and supportsDelayedJobs().

Referenced by push().

◆ deduplicateRootJob()

JobQueue::deduplicateRootJob ( IJobSpecification  $job)
final

Register the "root job" of a given job into the queue for de-duplication.

This should only be called right after all the new jobs have been inserted. This is used to turn older, duplicate, job entries into no-ops. The root job information will remain in the registry until it simply falls out of cache.

This requires that $job has two special fields in the "params" array:

  • rootJobSignature : hash (e.g. SHA1) that identifies the task
  • rootJobTimestamp : TS_MW timestamp of this instance of the task

A "root job" is a conceptual job that consist of potentially many smaller jobs that are actually inserted into the queue. For example, "refreshLinks" jobs are spawned when a template is edited. One can think of the task as "update links of pages that use template X" and an instance of that task as a "root job". However, what actually goes into the queue are range and leaf job subtypes. Since these jobs include things like page ID ranges and DB master positions, and can morph into smaller jobs recursively, simple duplicate detection for individual jobs being identical (like that of job_sha1) is not useful.

In the case of "refreshLinks", if these jobs are still in the queue when the template is edited again, we want all of these old refreshLinks jobs for that template to become no-ops. This can greatly reduce server load, since refreshLinks jobs involves parsing. Essentially, the new batch of jobs belong to a new "root job" and the older ones to a previous "root job" for the same task of "update links of pages that use template X".

This does nothing for certain queue classes.

Parameters
IJobSpecification$job
Exceptions
JobQueueError
Returns
bool

Definition at line 446 of file JobQueue.php.

References $type, assertNotReadOnly(), doDeduplicateRootJob(), and IJobSpecification\getType().

Referenced by batchPush().

◆ delayedJobsEnabled()

JobQueue::delayedJobsEnabled ( )
final
Returns
bool Whether delayed jobs are enabled
Since
1.22

Definition at line 175 of file JobQueue.php.

References supportsDelayedJobs().

◆ delete()

JobQueue::delete ( )
final

Deleted all unclaimed and delayed jobs from the queue.

Exceptions
JobQueueError
Since
1.22
Returns
void

Definition at line 538 of file JobQueue.php.

References assertNotReadOnly(), and doDelete().

◆ doAck()

JobQueue::doAck ( RunnableJob  $job)
abstractprotected
See also
JobQueue::ack()
Parameters
RunnableJob$job

Referenced by ack().

◆ doBatchPush()

JobQueue::doBatchPush ( array  $jobs,
  $flags 
)
abstractprotected
See also
JobQueue::batchPush()
Parameters
IJobSpecification[]$jobs
int$flags

Referenced by batchPush().

◆ doDeduplicateRootJob()

JobQueue::doDeduplicateRootJob ( IJobSpecification  $job)
protected

◆ doDelete()

JobQueue::doDelete ( )
protected
See also
JobQueue::delete()
Exceptions
JobQueueError

Definition at line 548 of file JobQueue.php.

Referenced by delete().

◆ doFlushCaches()

JobQueue::doFlushCaches ( )
protected
See also
JobQueue::flushCaches()
Returns
void

Definition at line 584 of file JobQueue.php.

References getAllQueuedJobs().

Referenced by flushCaches().

◆ doGetAbandonedCount()

JobQueue::doGetAbandonedCount ( )
protected
See also
JobQueue::getAbandonedCount()
Returns
int

Definition at line 296 of file JobQueue.php.

Referenced by getAbandonedCount().

◆ doGetAcquiredCount()

JobQueue::doGetAcquiredCount ( )
abstractprotected
See also
JobQueue::getAcquiredCount()
Returns
int

Referenced by getAcquiredCount().

◆ doGetDelayedCount()

JobQueue::doGetDelayedCount ( )
protected
See also
JobQueue::getDelayedCount()
Returns
int

Definition at line 273 of file JobQueue.php.

Referenced by getDelayedCount().

◆ doGetSiblingQueueSizes()

JobQueue::doGetSiblingQueueSizes ( array  $types)
protected
See also
JobQueue::getSiblingQueuesSize()
Parameters
array$typesList of queues types
Returns
array|null (list of queue types) or null if unsupported

Definition at line 685 of file JobQueue.php.

References null.

Referenced by getSiblingQueueSizes().

◆ doGetSiblingQueuesWithJobs()

JobQueue::doGetSiblingQueuesWithJobs ( array  $types)
protected
See also
JobQueue::getSiblingQueuesWithJobs()
Parameters
array$typesList of queues types
Returns
array|null (list of queue types) or null if unsupported

Definition at line 662 of file JobQueue.php.

References null.

Referenced by getSiblingQueuesWithJobs().

◆ doGetSize()

JobQueue::doGetSize ( )
abstractprotected
See also
JobQueue::getSize()
Returns
int

Referenced by getSize().

◆ doIsEmpty()

JobQueue::doIsEmpty ( )
abstractprotected
See also
JobQueue::isEmpty()
Returns
bool

Referenced by isEmpty().

◆ doIsRootJobOldDuplicate()

JobQueue::doIsRootJobOldDuplicate ( IJobSpecification  $job)
protected

◆ doPop()

JobQueue::doPop ( )
abstractprotected
See also
JobQueue::pop()
Returns
RunnableJob|bool

Referenced by pop().

◆ doWaitForBackups()

JobQueue::doWaitForBackups ( )
protected
See also
JobQueue::waitForBackups()
Returns
void

Definition at line 568 of file JobQueue.php.

Referenced by waitForBackups().

◆ factory()

static JobQueue::factory ( array  $params)
staticfinal

Get a job queue object of the specified type.

$params includes:

  • class : What job class to use (determines job type)
  • domain : Database domain ID of the wiki the jobs are for (defaults to current wiki)
  • type : The name of the job types this queue handles
  • order : Order that pop() selects jobs, one of "fifo", "timestamp" or "random". If "fifo" is used, the queue will effectively be FIFO. Note that job completion will not appear to be exactly FIFO if there are multiple job runners since jobs can take different times to finish once popped. If "timestamp" is used, the queue will at least be loosely ordered by timestamp, allowing for some jobs to be popped off out of order. If "random" is used, pop() will pick jobs in random order. Note that it may only be weakly random (e.g. a lottery of the oldest X). If "any" is choosen, the queue will use whatever order is the fastest. This might be useful for improving concurrency for job acquisition.
  • claimTTL : If supported, the queue will recycle jobs that have been popped but not acknowledged as completed after this many seconds. Recycling of jobs simply means re-inserting them into the queue. Jobs can be attempted up to three times before being discarded.
  • readOnlyReason : Set this to a string to make the queue read-only.
  • stash : A BagOStuff instance that can be used for root job deduplication
  • stats : A StatsdDataFactoryInterface [optional]

Queue classes should throw an exception if they do not support the options given.

Parameters
array$params
Returns
JobQueue
Exceptions
JobQueueError

Definition at line 106 of file JobQueue.php.

Referenced by JobQueueFederated\__construct(), CopyJobQueue\execute(), JobQueueGroup\get(), JobQueueGroup\getCoalescedQueues(), JobQueueMemoryTest\newJobQueue(), and JobQueueTest\setUp().

◆ factoryJob()

JobQueue::factoryJob (   $command,
  $params 
)
protected
Parameters
string$command
array$params
Returns
Job

Definition at line 694 of file JobQueue.php.

References $command, $params, and Job\factory().

Referenced by JobQueueRedis\getJobFromFields(), JobQueueRedis\getJobFromUidInternal(), JobQueueDB\jobFromRow(), and JobQueueMemory\jobFromSpecInternal().

◆ flushCaches()

JobQueue::flushCaches ( )
final

Clear any process and persistent caches.

Returns
void

Definition at line 576 of file JobQueue.php.

References doFlushCaches().

◆ getAbandonedCount()

JobQueue::getAbandonedCount ( )
final

Get the number of acquired jobs that can no longer be attempted.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns
int
Exceptions
JobQueueError

Definition at line 286 of file JobQueue.php.

References $res, and doGetAbandonedCount().

◆ getAcquiredCount()

JobQueue::getAcquiredCount ( )
final

Get the number of acquired jobs (these are temporarily out of the queue).

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns
int
Exceptions
JobQueueError

Definition at line 241 of file JobQueue.php.

References $res, and doGetAcquiredCount().

Referenced by JobQueueMemory\doAck().

◆ getAllAbandonedJobs()

JobQueue::getAllAbandonedJobs ( )

Get an iterator to traverse over all abandoned jobs in this queue.

Returns
Iterator
Exceptions
JobQueueError
Since
1.25

Definition at line 630 of file JobQueue.php.

Referenced by ManageJobs\repushAbandoned().

◆ getAllAcquiredJobs()

JobQueue::getAllAcquiredJobs ( )

Get an iterator to traverse over all claimed jobs in this queue.

Callers should be quick to iterator over it or few results will be returned due to jobs being acknowledged and deleted

Returns
Iterator
Exceptions
JobQueueError
Since
1.26

Definition at line 619 of file JobQueue.php.

◆ getAllDelayedJobs()

JobQueue::getAllDelayedJobs ( )

Get an iterator to traverse over all delayed jobs in this queue.

Note: results may be stale if the queue is concurrently modified.

Returns
Iterator
Exceptions
JobQueueError
Since
1.22

Definition at line 605 of file JobQueue.php.

◆ getAllQueuedJobs()

JobQueue::getAllQueuedJobs ( )
abstract

Get an iterator to traverse over all available jobs in this queue.

This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.

Returns
Iterator
Exceptions
JobQueueError

Referenced by doFlushCaches().

◆ getCoalesceLocationInternal()

JobQueue::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Returns
string
Since
1.22

Definition at line 640 of file JobQueue.php.

References null.

◆ getDelayedCount()

JobQueue::getDelayedCount ( )
final

Get the number of delayed jobs (these are temporarily out of the queue).

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns
int
Exceptions
JobQueueError
Since
1.22

Definition at line 263 of file JobQueue.php.

References $res, and doGetDelayedCount().

◆ getDomain()

JobQueue::getDomain ( )
final
Returns
string Database domain ID

Definition at line 122 of file JobQueue.php.

References $domain.

Referenced by ManageJobs\repushAbandoned().

◆ getOrder()

JobQueue::getOrder ( )
final
Returns
string One of (random, timestamp, fifo, undefined)

Definition at line 144 of file JobQueue.php.

References $order, optimalOrder(), and supportedOrders().

◆ getReadOnlyReason()

JobQueue::getReadOnlyReason ( )
Returns
string|bool Read-only rational or false if r/w
Since
1.27

Definition at line 183 of file JobQueue.php.

References $readOnlyReason.

◆ getRootJobCacheKey()

JobQueue::getRootJobCacheKey (   $signature)
protected
Parameters
string$signatureHash identifier of the root job
Returns
string

Definition at line 521 of file JobQueue.php.

References type.

Referenced by JobQueueRedis\doDeduplicateRootJob(), doDeduplicateRootJob(), JobQueueDB\doDeduplicateRootJob(), JobQueueRedis\doIsRootJobOldDuplicate(), and doIsRootJobOldDuplicate().

◆ getSiblingQueueSizes()

JobQueue::getSiblingQueueSizes ( array  $types)
final

Check the size of each of the given queues.

For queues not served by the same store as this one, 0 is returned. This is used for batching checks for queues stored at the same place.

Parameters
array$typesList of queues types
Returns
array|null (job type => whether queue is empty) or null if unsupported
Exceptions
JobQueueError
Since
1.22

Definition at line 676 of file JobQueue.php.

References doGetSiblingQueueSizes().

◆ getSiblingQueuesWithJobs()

JobQueue::getSiblingQueuesWithJobs ( array  $types)
final

Check whether each of the given queues are empty.

This is used for batching checks for queues stored at the same place.

Parameters
array$typesList of queues types
Returns
array|null (list of non-empty queue types) or null if unsupported
Exceptions
JobQueueError
Since
1.22

Definition at line 653 of file JobQueue.php.

References doGetSiblingQueuesWithJobs().

◆ getSize()

JobQueue::getSize ( )
final

Get the number of available (unacquired, non-delayed) jobs in the queue.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this number might be out of date for a minute.

Returns
int
Exceptions
JobQueueError

Definition at line 220 of file JobQueue.php.

References $res, and doGetSize().

◆ getType()

JobQueue::getType ( )
final
Returns
string Job type that this queue handles

Definition at line 137 of file JobQueue.php.

References $type.

Referenced by JobQueueDB\getAllAcquiredJobs(), JobQueueDB\getAllQueuedJobs(), and ManageJobs\repushAbandoned().

◆ getWiki()

JobQueue::getWiki ( )
final
Returns
string Wiki ID
Deprecated:
1.33

Definition at line 130 of file JobQueue.php.

References WikiMap\getWikiIdFromDbDomain().

◆ incrStats()

JobQueue::incrStats (   $key,
  $type,
  $delta = 1 
)
protected

Call wfIncrStats() for the queue overall and for the queue type.

Parameters
string$keyEvent type
string$typeJob type
int$delta
Since
1.22

Definition at line 716 of file JobQueue.php.

Referenced by JobQueueRedis\doAck(), JobQueueDB\doAck(), JobQueueRedis\doBatchPush(), JobQueueDB\doBatchPushInternal(), JobQueueDB\doPop(), JobQueueRedis\doPop(), pop(), and JobQueueDB\recycleAndDeleteStaleJobs().

◆ isEmpty()

JobQueue::isEmpty ( )
final

Quickly check if the queue has no available (unacquired, non-delayed) jobs.

Queue classes should use caching if they are any slower without memcached.

If caching is used, this might return false when there are actually no jobs. If pop() is called and returns false then it should correct the cache. Also, calling flushCaches() first prevents this. However, this affect is typically not distinguishable from the race condition between isEmpty() and pop().

Returns
bool
Exceptions
JobQueueError

Definition at line 199 of file JobQueue.php.

References $res, and doIsEmpty().

◆ isRootJobOldDuplicate()

JobQueue::isRootJobOldDuplicate ( IJobSpecification  $job)
finalprotected

Check if the "root" job of a given job has been superseded by a newer one.

Parameters
IJobSpecification$job
Exceptions
JobQueueError
Returns
bool

Definition at line 489 of file JobQueue.php.

References $type, doIsRootJobOldDuplicate(), and IJobSpecification\getType().

Referenced by pop().

◆ optimalOrder()

JobQueue::optimalOrder ( )
abstractprotected

Get the default queue order to use if configuration does not specify one.

Returns
string One of (random, timestamp, fifo, undefined)

Referenced by __construct(), and getOrder().

◆ pop()

JobQueue::pop ( )
final

Pop a job off of the queue.

This requires $wgJobClasses to be set for the given job type. Outside callers should use JobQueueGroup::pop() instead of this function.

Exceptions
JobQueueError
Returns
RunnableJob|bool Returns false if there are no jobs

Definition at line 366 of file JobQueue.php.

References $e, $job, assertNotReadOnly(), doPop(), incrStats(), isRootJobOldDuplicate(), DuplicateJob\newFromJob(), and type.

◆ push()

JobQueue::push (   $jobs,
  $flags = 0 
)
final

Push one or more jobs into the queue.

This does not require $wgJobClasses to be set for the given job type. Outside callers should use JobQueueGroup::push() instead of this function.

Parameters
IJobSpecification|IJobSpecification[]$jobs
int$flagsBitfield (supports JobQueue::QOS_ATOMIC)
Returns
void
Exceptions
JobQueueError

Definition at line 310 of file JobQueue.php.

References batchPush().

Referenced by CopyJobQueue\copyJobs(), and ManageJobs\repushAbandoned().

◆ supportedOrders()

JobQueue::supportedOrders ( )
abstractprotected

Get the allowed queue orders for configuration validation.

Returns
array Subset of (random, timestamp, fifo, undefined)

Referenced by __construct(), and getOrder().

◆ supportsDelayedJobs()

JobQueue::supportsDelayedJobs ( )
protected

Find out if delayed jobs are supported for configuration validation.

Returns
bool Whether delayed jobs are supported

Definition at line 167 of file JobQueue.php.

Referenced by batchPush(), and delayedJobsEnabled().

◆ waitForBackups()

JobQueue::waitForBackups ( )
final

Wait for any replica DBs or backup servers to catch up.

This does nothing for certain queue classes.

Returns
void
Exceptions
JobQueueError

Definition at line 560 of file JobQueue.php.

References doWaitForBackups().

Referenced by CopyJobQueue\copyJobs(), and ManageJobs\repushAbandoned().

Member Data Documentation

◆ $claimTTL

int JobQueue::$claimTTL
protected

Time to live in seconds.

Definition at line 39 of file JobQueue.php.

◆ $domain

string JobQueue::$domain
protected

DB domain ID.

Definition at line 33 of file JobQueue.php.

Referenced by JobQueueFederated\__construct(), getDomain(), and JobQueueMemory\getQueueData().

◆ $dupCache

BagOStuff JobQueue::$dupCache
protected

Definition at line 48 of file JobQueue.php.

Referenced by JobQueueDB\doDeduplicateRootJob().

◆ $maxTries

int JobQueue::$maxTries
protected

Maximum number of times to try a job.

Definition at line 41 of file JobQueue.php.

◆ $order

string JobQueue::$order
protected

Job priority for pop()

Definition at line 37 of file JobQueue.php.

Referenced by getOrder().

◆ $readOnlyReason

string bool JobQueue::$readOnlyReason
protected

Read only rationale (or false if r/w)

Definition at line 43 of file JobQueue.php.

Referenced by getReadOnlyReason().

◆ $stats

StatsdDataFactoryInterface JobQueue::$stats
protected

Definition at line 45 of file JobQueue.php.

◆ $type

◆ QOS_ATOMIC

const JobQueue::QOS_ATOMIC = 1

Definition at line 50 of file JobQueue.php.

◆ ROOTJOB_TTL

const JobQueue::ROOTJOB_TTL = 2419200

Definition at line 52 of file JobQueue.php.

Referenced by JobQueueDB\doDeduplicateRootJob().


The documentation for this class was generated from the following file: