MediaWiki  1.28.0
Public Member Functions | Protected Member Functions | Protected Attributes | Private Member Functions | List of all members
JobQueueRedis Class Reference
JobQueue | Redis

Class to handle job queues stored in Redis. More...

Inheritance diagram for JobQueueRedis:
Inheritance graph
[legend]
Collaboration diagram for JobQueueRedis:
Collaboration graph
[legend]

Public Member Functions

 __construct (array $params)
 
 getAllAbandonedJobs ()
 
 getAllAcquiredJobs ()
 
 getAllDelayedJobs ()
 
 getAllQueuedJobs ()
 
 getCoalesceLocationInternal ()
 
 getJobFromUidInternal ($uid, RedisConnRef $conn)
 This function should not be called outside JobQueueRedis. More...
 
 getServerQueuesWithJobs ()
 
- Public Member Functions inherited from JobQueue
 ack (Job $job)
 Acknowledge that a job was completed. More...
 
 batchPush (array $jobs, $flags=0)
 Push a batch of jobs into the queue. More...
 
 deduplicateRootJob (IJobSpecification $job)
 Register the "root job" of a given job into the queue for de-duplication. More...
 
 delayedJobsEnabled ()
 
 delete ()
 Deleted all unclaimed and delayed jobs from the queue. More...
 
 flushCaches ()
 Clear any process and persistent caches. More...
 
 getAbandonedCount ()
 Get the number of acquired jobs that can no longer be attempted. More...
 
 getAcquiredCount ()
 Get the number of acquired jobs (these are temporarily out of the queue). More...
 
 getAllAbandonedJobs ()
 Get an iterator to traverse over all abandoned jobs in this queue. More...
 
 getAllAcquiredJobs ()
 Get an iterator to traverse over all claimed jobs in this queue. More...
 
 getAllDelayedJobs ()
 Get an iterator to traverse over all delayed jobs in this queue. More...
 
 getAllQueuedJobs ()
 Get an iterator to traverse over all available jobs in this queue. More...
 
 getCoalesceLocationInternal ()
 Do not use this function outside of JobQueue/JobQueueGroup. More...
 
 getDelayedCount ()
 Get the number of delayed jobs (these are temporarily out of the queue). More...
 
 getOrder ()
 
 getReadOnlyReason ()
 
 getSiblingQueueSizes (array $types)
 Check the size of each of the given queues. More...
 
 getSiblingQueuesWithJobs (array $types)
 Check whether each of the given queues are empty. More...
 
 getSize ()
 Get the number of available (unacquired, non-delayed) jobs in the queue. More...
 
 getType ()
 
 getWiki ()
 
 isEmpty ()
 Quickly check if the queue has no available (unacquired, non-delayed) jobs. More...
 
 pop ()
 Pop a job off of the queue. More...
 
 push ($jobs, $flags=0)
 Push one or more jobs into the queue. More...
 
 waitForBackups ()
 Wait for any replica DBs or backup servers to catch up. More...
 

Protected Member Functions

 doAck (Job $job)
 
 doBatchPush (array $jobs, $flags)
 
 doDeduplicateRootJob (IJobSpecification $job)
 
 doDelete ()
 
 doGetAbandonedCount ()
 
 doGetAcquiredCount ()
 
 doGetDelayedCount ()
 
 doGetSiblingQueueSizes (array $types)
 
 doGetSiblingQueuesWithJobs (array $types)
 
 doGetSize ()
 
 doIsEmpty ()
 
 doIsRootJobOldDuplicate (Job $job)
 
 doPop ()
 
 getConnection ()
 Get a connection to the server that handles all sub-queues for this queue. More...
 
 getJobFromFields (array $fields)
 
 getJobIterator (RedisConnRef $conn, array $uids)
 
 getNewJobFields (IJobSpecification $job)
 
 optimalOrder ()
 
 popAndAcquireBlob (RedisConnRef $conn)
 
 pushBlobs (RedisConnRef $conn, array $items)
 
 serialize (array $fields)
 
 supportedOrders ()
 
 supportsDelayedJobs ()
 
 throwRedisException (RedisConnRef $conn, $e)
 
 unserialize ($blob)
 
- Protected Member Functions inherited from JobQueue
 __construct (array $params)
 
 assertNotReadOnly ()
 
 doAck (Job $job)
 
 doBatchPush (array $jobs, $flags)
 
 doDeduplicateRootJob (IJobSpecification $job)
 
 doDelete ()
 
 doFlushCaches ()
 
 doGetAbandonedCount ()
 
 doGetAcquiredCount ()
 
 doGetDelayedCount ()
 
 doGetSiblingQueueSizes (array $types)
 
 doGetSiblingQueuesWithJobs (array $types)
 
 doGetSize ()
 
 doIsEmpty ()
 
 doIsRootJobOldDuplicate (Job $job)
 
 doPop ()
 
 doWaitForBackups ()
 
 getRootJobCacheKey ($signature)
 
 isRootJobOldDuplicate (Job $job)
 Check if the "root" job of a given job has been superseded by a newer one. More...
 
 optimalOrder ()
 Get the default queue order to use if configuration does not specify one. More...
 
 supportedOrders ()
 Get the allowed queue orders for configuration validation. More...
 
 supportsDelayedJobs ()
 Find out if delayed jobs are supported for configuration validation. More...
 

Protected Attributes

string $compression
 Compression method to use. More...
 
LoggerInterface $logger
 
RedisConnectionPool $redisPool
 
string $server
 Server address. More...
 
- Protected Attributes inherited from JobQueue
JobQueueAggregator $aggr
 
int $claimTTL
 Time to live in seconds. More...
 
BagOStuff $dupCache
 
int $maxTries
 Maximum number of times to try a job. More...
 
string $order
 Job priority for pop() More...
 
string bool $readOnlyReason
 Read only rationale (or false if r/w) More...
 
string $type
 Job type. More...
 
string $wiki
 Wiki ID. More...
 

Private Member Functions

 decodeQueueName ($name)
 
 encodeQueueName ()
 
 getGlobalKey ($name)
 
 getQueueKey ($prop, $type=null)
 

Additional Inherited Members

- Static Public Member Functions inherited from JobQueue
static factory (array $params)
 Get a job queue object of the specified type. More...
 
static incrStats ($key, $type, $delta=1)
 Call wfIncrStats() for the queue overall and for the queue type. More...
 
- Public Attributes inherited from JobQueue
const QOS_ATOMIC = 1
 
const ROOTJOB_TTL = 2419200
 

Detailed Description

Class to handle job queues stored in Redis.

This is a faster and less resource-intensive job queue than JobQueueDB. All data for a queue using this class is placed into one redis server. The mediawiki/services/jobrunner background service must be set up and running.

There are eight main redis keys (per queue) used to track jobs:

  • l-unclaimed : A list of job IDs used for ready unclaimed jobs
  • z-claimed : A sorted set of (job ID, UNIX timestamp as score) used for job retries
  • z-abandoned : A sorted set of (job ID, UNIX timestamp as score) used for broken jobs
  • z-delayed : A sorted set of (job ID, UNIX timestamp as score) used for delayed jobs
  • h-idBySha1 : A hash of (SHA1 => job ID) for unclaimed jobs used for de-duplication
  • h-sha1ById : A hash of (job ID => SHA1) for unclaimed jobs used for de-duplication
  • h-attempts : A hash of (job ID => attempt count) used for job claiming/retries
  • h-data : A hash of (job ID => serialized blobs) for job storage A job ID can be in only one of z-delayed, l-unclaimed, z-claimed, and z-abandoned. If an ID appears in any of those lists, it should have a h-data entry for its ID. If a job has a SHA1 de-duplication value and its ID is in l-unclaimed or z-delayed, then there should be no other such jobs with that SHA1. Every h-idBySha1 entry has an h-sha1ById entry and every h-sha1ById must refer to an ID that is l-unclaimed. If a job has its ID in z-claimed or z-abandoned, then it must also have an h-attempts entry for its ID.

The following keys are used to track queue states:

  • s-queuesWithJobs : A set of all queues with non-abandoned jobs

The background service takes care of undelaying, recycling, and pruning jobs as well as removing s-queuesWithJobs entries as queues empty.

Additionally, "rootjob:* keys track "root jobs" used for additional de-duplication. Aside from root job keys, all keys have no expiry, and are only removed when jobs are run. All the keys are prefixed with the relevant wiki ID information.

This class requires Redis 2.6 as it makes use Lua scripts for fast atomic operations. Additionally, it should be noted that redis has different persistence modes, such as rdb snapshots, journaling, and no persistence. Appropriate configuration should be made on the servers based on what queues are using it and what tolerance they have.

Since
1.22

Definition at line 67 of file JobQueueRedis.php.

Constructor & Destructor Documentation

JobQueueRedis::__construct ( array  $params)
Parameters
array$paramsPossible keys:
  • redisConfig : An array of parameters to RedisConnectionPool::__construct(). Note that the serializer option is ignored as "none" is always used.
  • redisServer : A hostname/port combination or the absolute path of a UNIX socket. If a hostname is specified but no port, the standard port number 6379 will be used. Required.
  • compression : The type of compression to use; one of (none,gzip).
  • daemonized : Set to true if the redisJobRunnerService runs in the background. This will disable job recycling/undelaying from the MediaWiki side to avoid redundance and out-of-sync configuration.
Exceptions
InvalidArgumentException

Definition at line 91 of file JobQueueRedis.php.

References MediaWiki\Logger\LoggerFactory\getInstance(), server, and RedisConnectionPool\singleton().

Member Function Documentation

JobQueueRedis::decodeQueueName (   $name)
private
Parameters
string$nameJSON
Returns
array (type, wiki)

Definition at line 785 of file JobQueueRedis.php.

References $name.

Referenced by getServerQueuesWithJobs().

JobQueueRedis::doAck ( Job  $job)
protected
See also
JobQueue::doAck()
Parameters
Job$job
Returns
Job|bool
Exceptions
UnexpectedValueException
JobQueueError

Definition at line 386 of file JobQueueRedis.php.

References $e, $res, are, as, data, getConnection(), getQueueKey(), JobQueue\incrStats(), of, that, the, then, throwRedisException(), type, and wfDebugLog().

JobQueueRedis::doBatchPush ( array  $jobs,
  $flags 
)
protected
See also
JobQueue::doBatchPush()
Parameters
IJobSpecification[]$jobs
int$flags
Returns
void
Exceptions
JobQueueError

Definition at line 193 of file JobQueueRedis.php.

References $e, $flags, $job, as, getConnection(), getNewJobFields(), JobQueue\incrStats(), pushBlobs(), throwRedisException(), type, and wfDebugLog().

JobQueueRedis::doDeduplicateRootJob ( IJobSpecification  $job)
protected
See also
JobQueue::doDeduplicateRootJob()
Parameters
IJobSpecification$job
Returns
bool
Exceptions
JobQueueError
LogicException

Definition at line 441 of file JobQueueRedis.php.

References $e, $params, $timestamp, getConnection(), JobQueue\getRootJobCacheKey(), IJobSpecification\getRootJobParams(), IJobSpecification\hasRootJobParams(), and throwRedisException().

JobQueueRedis::doDelete ( )
protected
See also
JobQueue::doDelete()
Returns
bool
Exceptions
JobQueueError

Definition at line 493 of file JobQueueRedis.php.

References $e, $keys, as, encodeQueueName(), false, getConnection(), getGlobalKey(), getQueueKey(), and throwRedisException().

JobQueueRedis::doGetAbandonedCount ( )
protected
See also
JobQueue::doGetAbandonedCount()
Returns
int
Exceptions
JobQueueError

Definition at line 177 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

JobQueueRedis::doGetAcquiredCount ( )
protected
See also
JobQueue::doGetAcquiredCount()
Returns
int
Exceptions
JobQueueError

Definition at line 145 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

JobQueueRedis::doGetDelayedCount ( )
protected
See also
JobQueue::doGetDelayedCount()
Returns
int
Exceptions
JobQueueError

Definition at line 163 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

JobQueueRedis::doGetSiblingQueueSizes ( array  $types)
protected

Definition at line 602 of file JobQueueRedis.php.

References $e, $res, JobQueue\$type, as, getConnection(), getQueueKey(), and throwRedisException().

Referenced by doGetSiblingQueuesWithJobs().

JobQueueRedis::doGetSiblingQueuesWithJobs ( array  $types)
protected

Definition at line 598 of file JobQueueRedis.php.

References doGetSiblingQueueSizes().

JobQueueRedis::doGetSize ( )
protected
See also
JobQueue::doGetSize()
Returns
int
Exceptions
JobQueueError

Definition at line 131 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

Referenced by doIsEmpty().

JobQueueRedis::doIsEmpty ( )
protected
See also
JobQueue::doIsEmpty()
Returns
bool
Exceptions
JobQueueError

Definition at line 122 of file JobQueueRedis.php.

References doGetSize().

JobQueueRedis::doIsRootJobOldDuplicate ( Job  $job)
protected
See also
JobQueue::doIsRootJobOldDuplicate()
Parameters
Job$job
Returns
bool
Exceptions
JobQueueError

Definition at line 469 of file JobQueueRedis.php.

References $e, $params, $timestamp, getConnection(), JobQueue\getRootJobCacheKey(), Job\getRootJobParams(), Job\hasRootJobParams(), and throwRedisException().

JobQueueRedis::doPop ( )
protected
See also
JobQueue::doPop()
Returns
Job|bool
Exceptions
JobQueueError

Definition at line 312 of file JobQueueRedis.php.

References $blob, $e, $job, getConnection(), getJobFromFields(), JobQueue\incrStats(), popAndAcquireBlob(), throwRedisException(), type, unserialize(), and wfDebugLog().

JobQueueRedis::encodeQueueName ( )
private
Returns
string JSON

Definition at line 777 of file JobQueueRedis.php.

References type, and wiki.

Referenced by doDelete(), and pushBlobs().

JobQueueRedis::getAllAbandonedJobs ( )
See also
JobQueue::getAllAbandonedJobs()
Returns
Iterator
Exceptions
JobQueueError

Definition at line 566 of file JobQueueRedis.php.

References $e, getConnection(), getJobIterator(), getQueueKey(), and throwRedisException().

JobQueueRedis::getAllAcquiredJobs ( )
See also
JobQueue::getAllAcquiredJobs()
Returns
Iterator
Exceptions
JobQueueError

Definition at line 550 of file JobQueueRedis.php.

References $e, getConnection(), getJobIterator(), getQueueKey(), and throwRedisException().

JobQueueRedis::getAllDelayedJobs ( )
See also
JobQueue::getAllDelayedJobs()
Returns
Iterator
Exceptions
JobQueueError

Definition at line 534 of file JobQueueRedis.php.

References $e, getConnection(), getJobIterator(), getQueueKey(), and throwRedisException().

JobQueueRedis::getAllQueuedJobs ( )
See also
JobQueue::getAllQueuedJobs()
Returns
Iterator
Exceptions
JobQueueError

Definition at line 518 of file JobQueueRedis.php.

References $e, getConnection(), getJobIterator(), getQueueKey(), and throwRedisException().

JobQueueRedis::getCoalesceLocationInternal ( )

Definition at line 594 of file JobQueueRedis.php.

References $server.

JobQueueRedis::getConnection ( )
protected

Get a connection to the server that handles all sub-queues for this queue.

Returns
RedisConnRef
Exceptions
JobQueueConnectionError

Definition at line 754 of file JobQueueRedis.php.

References server.

Referenced by doAck(), doBatchPush(), doDeduplicateRootJob(), doDelete(), doGetAbandonedCount(), doGetAcquiredCount(), doGetDelayedCount(), doGetSiblingQueueSizes(), doGetSize(), doIsRootJobOldDuplicate(), doPop(), getAllAbandonedJobs(), getAllAcquiredJobs(), getAllDelayedJobs(), getAllQueuedJobs(), and getServerQueuesWithJobs().

JobQueueRedis::getGlobalKey (   $name)
private
Parameters
string$name
Returns
string

Definition at line 793 of file JobQueueRedis.php.

References $name, and as.

Referenced by doDelete(), getServerQueuesWithJobs(), and pushBlobs().

JobQueueRedis::getJobFromFields ( array  $fields)
protected
Parameters
array$fields
Returns
Job|bool

Definition at line 703 of file JobQueueRedis.php.

References $job, $title, Job\factory(), and Title\makeTitle().

Referenced by doPop().

JobQueueRedis::getJobFromUidInternal (   $uid,
RedisConnRef  $conn 
)

This function should not be called outside JobQueueRedis.

Parameters
string$uid
RedisConnRef$conn
Returns
Job|bool Returns false if the job does not exist
Exceptions
JobQueueError
UnexpectedValueException

Definition at line 633 of file JobQueueRedis.php.

References $e, $job, $title, Job\factory(), getQueueKey(), Title\makeTitle(), throwRedisException(), and unserialize().

Referenced by getJobIterator().

JobQueueRedis::getJobIterator ( RedisConnRef  $conn,
array  $uids 
)
protected
Parameters
RedisConnRef$conn
array$uidsList of job UUIDs
Returns
MappedIterator

Definition at line 582 of file JobQueueRedis.php.

References $job, getJobFromUidInternal(), and use.

Referenced by getAllAbandonedJobs(), getAllAcquiredJobs(), getAllDelayedJobs(), and getAllQueuedJobs().

JobQueueRedis::getNewJobFields ( IJobSpecification  $job)
protected
Parameters
IJobSpecification$job
Returns
array

Definition at line 681 of file JobQueueRedis.php.

References IJobSpecification\getDeduplicationInfo(), IJobSpecification\getParams(), IJobSpecification\getReleaseTimestamp(), IJobSpecification\getTitle(), IJobSpecification\getType(), IJobSpecification\ignoreDuplicates(), UIDGenerator\newRawUUIDv4(), UIDGenerator\QUICK_RAND, and serialize().

Referenced by doBatchPush().

JobQueueRedis::getQueueKey (   $prop,
  $type = null 
)
private
Parameters
string$prop
string | null$typeOverride this for sibling queues
Returns
string

Definition at line 809 of file JobQueueRedis.php.

References JobQueue\$type, list, wfSplitWikiID(), and wiki.

Referenced by doAck(), doDelete(), doGetAbandonedCount(), doGetAcquiredCount(), doGetDelayedCount(), doGetSiblingQueueSizes(), doGetSize(), getAllAbandonedJobs(), getAllAcquiredJobs(), getAllDelayedJobs(), getAllQueuedJobs(), getJobFromUidInternal(), popAndAcquireBlob(), and pushBlobs().

JobQueueRedis::getServerQueuesWithJobs ( )
Returns
array List of (wiki,type) tuples for queues with non-abandoned jobs
Exceptions
JobQueueConnectionError
JobQueueError

Definition at line 661 of file JobQueueRedis.php.

References $e, $queue, as, decodeQueueName(), getConnection(), getGlobalKey(), and throwRedisException().

JobQueueRedis::optimalOrder ( )
protected

Definition at line 109 of file JobQueueRedis.php.

JobQueueRedis::popAndAcquireBlob ( RedisConnRef  $conn)
protected
Parameters
RedisConnRef$conn
Returns
array Serialized string or false
Exceptions
RedisException

Definition at line 345 of file JobQueueRedis.php.

References and(), are, as, getQueueKey(), RedisConnRef\luaEval(), not, of, that, the, and then.

Referenced by doPop().

JobQueueRedis::pushBlobs ( RedisConnRef  $conn,
array  $items 
)
protected
Parameters
RedisConnRef$conn
array$itemsList of results from JobQueueRedis::getNewJobFields()
Returns
int Number of jobs inserted (duplicates are ignored)
Exceptions
RedisException

Definition at line 247 of file JobQueueRedis.php.

References $args, are, as, encodeQueueName(), First, getGlobalKey(), getQueueKey(), id, in, is(), RedisConnRef\luaEval(), needed(), of, or, release, serialize(), string, that, the, and then.

Referenced by doBatchPush().

JobQueueRedis::serialize ( array  $fields)
protected
Parameters
array$fields
Returns
string Serialized and possibly compressed version of $fields

Definition at line 716 of file JobQueueRedis.php.

References $blob, and object.

Referenced by getNewJobFields(), and pushBlobs().

JobQueueRedis::supportedOrders ( )
protected

Definition at line 105 of file JobQueueRedis.php.

JobQueueRedis::supportsDelayedJobs ( )
protected

Definition at line 113 of file JobQueueRedis.php.

JobQueueRedis::throwRedisException ( RedisConnRef  $conn,
  $e 
)
protected
Parameters
RedisConnRef$conn
RedisException$e
Exceptions
JobQueueError

Definition at line 769 of file JobQueueRedis.php.

References $e.

Referenced by doAck(), doBatchPush(), doDeduplicateRootJob(), doDelete(), doGetAbandonedCount(), doGetAcquiredCount(), doGetDelayedCount(), doGetSiblingQueueSizes(), doGetSize(), doIsRootJobOldDuplicate(), doPop(), getAllAbandonedJobs(), getAllAcquiredJobs(), getAllDelayedJobs(), getAllQueuedJobs(), getJobFromUidInternal(), and getServerQueuesWithJobs().

JobQueueRedis::unserialize (   $blob)
protected
Parameters
string$blob
Returns
array|bool Unserialized version of $blob or false

Definition at line 735 of file JobQueueRedis.php.

References $blob.

Referenced by doPop(), and getJobFromUidInternal().

Member Data Documentation

string JobQueueRedis::$compression
protected

Compression method to use.

Definition at line 76 of file JobQueueRedis.php.

LoggerInterface JobQueueRedis::$logger
protected

Definition at line 71 of file JobQueueRedis.php.

RedisConnectionPool JobQueueRedis::$redisPool
protected

Definition at line 69 of file JobQueueRedis.php.

string JobQueueRedis::$server
protected

Server address.

Definition at line 74 of file JobQueueRedis.php.

Referenced by getCoalesceLocationInternal().

The documentation for this class was generated from the following file: