MediaWiki  1.23.2
JobQueueRedis Class Reference

Class to handle job queues stored in Redis. More...

Inheritance diagram for JobQueueRedis:
Collaboration diagram for JobQueueRedis:

Public Member Functions

 __construct (array $params)
 @params include: More...
 
 getAllDelayedJobs ()
 
 getAllQueuedJobs ()
 
 getCoalesceLocationInternal ()
 Do not use this function outside of JobQueue/JobQueueGroup. More...
 
 getJobFromUidInternal ( $uid, RedisConnRef $conn)
 This function should not be called outside JobQueueRedis. More...
 
 recyclePruneAndUndelayJobs ()
 Recycle or destroy any jobs that have been claimed for too long and release any ready delayed jobs into the queue. More...
 
 setTestingPrefix ( $key)
 
- Public Member Functions inherited from JobQueue
 ack (Job $job)
 Acknowledge that a job was completed. More...
 
 batchPush (array $jobs, $flags=0)
 Push a batch of jobs into the queue. More...
 
 deduplicateRootJob (Job $job)
 Register the "root job" of a given job into the queue for de-duplication. More...
 
 delayedJobsEnabled ()
 
 delete ()
 Deleted all unclaimed and delayed jobs from the queue. More...
 
 flushCaches ()
 Clear any process and persistent caches. More...
 
 getAbandonedCount ()
 Get the number of acquired jobs that can no longer be attempted. More...
 
 getAcquiredCount ()
 Get the number of acquired jobs (these are temporarily out of the queue). More...
 
 getDelayedCount ()
 Get the number of delayed jobs (these are temporarily out of the queue). More...
 
 getOrder ()
 
 getPeriodicTasks ()
 Return a map of task names to task definition maps. More...
 
 getSiblingQueueSizes (array $types)
 Check the size of each of the given queues. More...
 
 getSiblingQueuesWithJobs (array $types)
 Check whether each of the given queues are empty. More...
 
 getSize ()
 Get the number of available (unacquired, non-delayed) jobs in the queue. More...
 
 getType ()
 
 getWiki ()
 
 isEmpty ()
 Quickly check if the queue has no available (unacquired, non-delayed) jobs. More...
 
 pop ()
 Pop a job off of the queue. More...
 
 push ( $jobs, $flags=0)
 Push one or more jobs into the queue. More...
 
 waitForBackups ()
 Wait for any slaves or backup servers to catch up. More...
 

Public Attributes

const MAX_AGE_PRUNE = 604800
 
- Public Attributes inherited from JobQueue
const QOS_ATOMIC = 1
 
const ROOTJOB_TTL = 2419200
 

Protected Member Functions

 doAck (Job $job)
 
 doBatchPush (array $jobs, $flags)
 
 doDeduplicateRootJob (Job $job)
 
 doDelete ()
 
 doGetAbandonedCount ()
 
 doGetAcquiredCount ()
 
 doGetDelayedCount ()
 
 doGetPeriodicTasks ()
 
 doGetSiblingQueueSizes (array $types)
 
 doGetSiblingQueuesWithJobs (array $types)
 
 doGetSize ()
 
 doIsEmpty ()
 
 doIsRootJobOldDuplicate (Job $job)
 
 doPop ()
 
 getConnection ()
 Get a connection to the server that handles all sub-queues for this queue. More...
 
 getJobFromFields (array $fields)
 
 getNewJobFields (IJobSpecification $job)
 
 optimalOrder ()
 Get the default queue order to use if configuration does not specify one. More...
 
 popAndAcquireBlob (RedisConnRef $conn)
 
 popAndDeleteBlob (RedisConnRef $conn)
 
 pushBlobs (RedisConnRef $conn, array $items)
 
 serialize (array $fields)
 
 supportedOrders ()
 Get the allowed queue orders for configuration validation. More...
 
 supportsDelayedJobs ()
 Find out if delayed jobs are supported for configuration validation. More...
 
 throwRedisException (RedisConnRef $conn, $e)
 
 unserialize ( $blob)
 
- Protected Member Functions inherited from JobQueue
 doFlushCaches ()
 
 doWaitForBackups ()
 
 getRootJobCacheKey ( $signature)
 
 isRootJobOldDuplicate (Job $job)
 Check if the "root" job of a given job has been superseded by a newer one. More...
 

Protected Attributes

string $compression
 Compression method to use *. More...
 
string $key
 Key to prefix the queue keys with (used for testing) *. More...
 
null int $maximumPeriodicTaskSeconds
 maximum seconds between execution of periodic tasks. More...
 
RedisConnectionPool $redisPool
 
string $server
 Server address *. More...
 
- Protected Attributes inherited from JobQueue
bool $checkDelay
 Allow delayed jobs *. More...
 
int $claimTTL
 Time to live in seconds *. More...
 
BagOStuff $dupCache
 
int $maxTries
 Maximum number of times to try a job *. More...
 
string $order
 Job priority for pop() *. More...
 
string $type
 Job type *. More...
 
string $wiki
 Wiki ID *. More...
 

Private Member Functions

 getQueueKey ( $prop, $type=null)
 

Additional Inherited Members

- Static Public Member Functions inherited from JobQueue
static factory (array $params)
 Get a job queue object of the specified type. More...
 
static incrStats ( $key, $type, $delta=1, $wiki=null)
 Call wfIncrStats() for the queue overall and for the queue type. More...
 

Detailed Description

Class to handle job queues stored in Redis.

This is faster, less resource intensive, queue that JobQueueDB. All data for a queue using this class is placed into one redis server.

There are eight main redis keys used to track jobs:

  • l-unclaimed : A list of job IDs used for ready unclaimed jobs
  • z-claimed : A sorted set of (job ID, UNIX timestamp as score) used for job retries
  • z-abandoned : A sorted set of (job ID, UNIX timestamp as score) used for broken jobs
  • z-delayed : A sorted set of (job ID, UNIX timestamp as score) used for delayed jobs
  • h-idBySha1 : A hash of (SHA1 => job ID) for unclaimed jobs used for de-duplication
  • h-sha1ById : A hash of (job ID => SHA1) for unclaimed jobs used for de-duplication
  • h-attempts : A hash of (job ID => attempt count) used for job claiming/retries
  • h-data : A hash of (job ID => serialized blobs) for job storage A job ID can be in only one of z-delayed, l-unclaimed, z-claimed, and z-abandoned. If an ID appears in any of those lists, it should have a h-data entry for its ID. If a job has a SHA1 de-duplication value and its ID is in l-unclaimed or z-delayed, then there should be no other such jobs with that SHA1. Every h-idBySha1 entry has an h-sha1ById entry and every h-sha1ById must refer to an ID that is l-unclaimed. If a job has its ID in z-claimed or z-abandoned, then it must also have an h-attempts entry for its ID.

Additionally, "rootjob:* keys track "root jobs" used for additional de-duplication. Aside from root job keys, all keys have no expiry, and are only removed when jobs are run. All the keys are prefixed with the relevant wiki ID information.

This class requires Redis 2.6 as it makes use Lua scripts for fast atomic operations. Additionally, it should be noted that redis has different persistence modes, such as rdb snapshots, journaling, and no persistent. Appropriate configuration should be made on the servers based on what queues are using it and what tolerance they have.

Since
1.22

Definition at line 59 of file JobQueueRedis.php.

Constructor & Destructor Documentation

◆ __construct()

JobQueueRedis::__construct ( array  $params)

@params include:

  • redisConfig : An array of parameters to RedisConnectionPool::__construct(). Note that the serializer option is ignored as "none" is always used.
  • redisServer : A hostname/port combination or the absolute path of a UNIX socket. If a hostname is specified but no port, the standard port number 6379 will be used. Required.
  • compression : The type of compression to use; one of (none,gzip).
  • maximumPeriodicTaskSeconds : Maximum seconds between check periodic tasks. Set to force faster execution of periodic tasks for inegration tests that rely on checkDelay. Without this the integration tests are very very slow. This really shouldn't be set in production.
    Parameters
    array$params

Reimplemented from JobQueue.

Definition at line 89 of file JobQueueRedis.php.

References $params, server, and RedisConnectionPool\singleton().

Member Function Documentation

◆ doAck()

JobQueueRedis::doAck ( Job  $job)
protected
See also
JobQueue::doAck()
Parameters
Job$job
Returns
Job|bool
Exceptions
MWException|JobQueueError

Reimplemented from JobQueue.

Definition at line 420 of file JobQueueRedis.php.

References $e, $job, $res, are, array(), as, data, getConnection(), getQueueKey(), of, that, throwRedisException(), and wfDebugLog().

◆ doBatchPush()

JobQueueRedis::doBatchPush ( array  $jobs,
  $flags 
)
protected
See also
JobQueue::doBatchPush()
Parameters
array$jobs
$flags
Returns
bool
Exceptions
JobQueueError

Reimplemented from JobQueue.

Definition at line 196 of file JobQueueRedis.php.

References $e, $failed, $flags, $job, array(), as, getConnection(), getNewJobFields(), JobQueue\incrStats(), pushBlobs(), throwRedisException(), type, wfDebugLog(), and wiki.

◆ doDeduplicateRootJob()

JobQueueRedis::doDeduplicateRootJob ( Job  $job)
protected
See also
JobQueue::doDeduplicateRootJob()
Parameters
Job$job
Returns
bool
Exceptions
MWException|JobQueueError

Reimplemented from JobQueue.

Definition at line 465 of file JobQueueRedis.php.

References $e, $job, $key, $params, $timestamp, getConnection(), JobQueue\getRootJobCacheKey(), and throwRedisException().

◆ doDelete()

JobQueueRedis::doDelete ( )
protected
See also
JobQueue::doDelete()
Returns
bool
Exceptions
JobQueueError

Reimplemented from JobQueue.

Definition at line 516 of file JobQueueRedis.php.

References $e, $keys, array(), as, false, getConnection(), getQueueKey(), and throwRedisException().

◆ doGetAbandonedCount()

JobQueueRedis::doGetAbandonedCount ( )
protected
See also
JobQueue::doGetAbandonedCount()
Returns
int
Exceptions
JobQueueError

Reimplemented from JobQueue.

Definition at line 177 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

◆ doGetAcquiredCount()

JobQueueRedis::doGetAcquiredCount ( )
protected
See also
JobQueue::doGetAcquiredCount()
Returns
int
Exceptions
JobQueueError

Reimplemented from JobQueue.

Definition at line 139 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

◆ doGetDelayedCount()

JobQueueRedis::doGetDelayedCount ( )
protected
See also
JobQueue::doGetDelayedCount()
Returns
int
Exceptions
JobQueueError

Reimplemented from JobQueue.

Definition at line 160 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

◆ doGetPeriodicTasks()

JobQueueRedis::doGetPeriodicTasks ( )
protected
Returns
array

Reimplemented from JobQueue.

Definition at line 726 of file JobQueueRedis.php.

References array().

◆ doGetSiblingQueueSizes()

JobQueueRedis::doGetSiblingQueueSizes ( array  $types)
protected
See also
JobQueue::getSiblingQueuesSize()
Parameters
array$typesList of queues types
Returns
array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 587 of file JobQueueRedis.php.

References $e, $res, $size, JobQueue\$type, array(), as, getConnection(), getQueueKey(), and throwRedisException().

Referenced by doGetSiblingQueuesWithJobs().

◆ doGetSiblingQueuesWithJobs()

JobQueueRedis::doGetSiblingQueuesWithJobs ( array  $types)
protected
See also
JobQueue::getSiblingQueuesWithJobs()
Parameters
array$typesList of queues types
Returns
array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 583 of file JobQueueRedis.php.

References doGetSiblingQueueSizes().

◆ doGetSize()

JobQueueRedis::doGetSize ( )
protected
See also
JobQueue::doGetSize()
Returns
int
Exceptions
MWException

Reimplemented from JobQueue.

Definition at line 125 of file JobQueueRedis.php.

References $e, getConnection(), getQueueKey(), and throwRedisException().

Referenced by doIsEmpty().

◆ doIsEmpty()

JobQueueRedis::doIsEmpty ( )
protected
See also
JobQueue::doIsEmpty()
Returns
bool
Exceptions
MWException

Reimplemented from JobQueue.

Definition at line 116 of file JobQueueRedis.php.

References doGetSize().

◆ doIsRootJobOldDuplicate()

JobQueueRedis::doIsRootJobOldDuplicate ( Job  $job)
protected
See also
JobQueue::doIsRootJobOldDuplicate()
Parameters
Job$job
Returns
bool
Exceptions
JobQueueError

Reimplemented from JobQueue.

Definition at line 493 of file JobQueueRedis.php.

References $e, $job, $params, $timestamp, getConnection(), JobQueue\getRootJobCacheKey(), and throwRedisException().

◆ doPop()

JobQueueRedis::doPop ( )
protected

◆ getAllDelayedJobs()

JobQueueRedis::getAllDelayedJobs ( )
See also
JobQueue::getAllQueuedJobs()
Returns
Iterator

Reimplemented from JobQueue.

Definition at line 560 of file JobQueueRedis.php.

References $e, $job, array(), getConnection(), and throwRedisException().

◆ getAllQueuedJobs()

JobQueueRedis::getAllQueuedJobs ( )
See also
JobQueue::getAllQueuedJobs()
Returns
Iterator

Reimplemented from JobQueue.

Definition at line 537 of file JobQueueRedis.php.

References $e, $job, array(), getConnection(), and throwRedisException().

◆ getCoalesceLocationInternal()

JobQueueRedis::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Returns
string
Since
1.22

Reimplemented from JobQueue.

Definition at line 579 of file JobQueueRedis.php.

References $server.

◆ getConnection()

JobQueueRedis::getConnection ( )
protected

◆ getJobFromFields()

JobQueueRedis::getJobFromFields ( array  $fields)
protected
Parameters
$fieldsarray
Returns
Job|bool

Definition at line 774 of file JobQueueRedis.php.

References $job, $title, Job\factory(), and Title\makeTitleSafe().

Referenced by doPop().

◆ getJobFromUidInternal()

JobQueueRedis::getJobFromUidInternal (   $uid,
RedisConnRef  $conn 
)

This function should not be called outside JobQueueRedis.

Parameters
$uidstring
$connRedisConnRef
Returns
Job|bool Returns false if the job does not exist
Exceptions
MWException|JobQueueError

Definition at line 617 of file JobQueueRedis.php.

References $e, $job, $title, Job\factory(), getQueueKey(), Title\makeTitle(), throwRedisException(), and unserialize().

◆ getNewJobFields()

JobQueueRedis::getNewJobFields ( IJobSpecification  $job)
protected
Parameters
IJobSpecification$job
Returns
array

Definition at line 752 of file JobQueueRedis.php.

References $job, array(), UIDGenerator\newRawUUIDv4(), UIDGenerator\QUICK_RAND, serialize(), and wfBaseConvert().

Referenced by doBatchPush().

◆ getQueueKey()

JobQueueRedis::getQueueKey (   $prop,
  $type = null 
)
private

◆ optimalOrder()

JobQueueRedis::optimalOrder ( )
protected

Get the default queue order to use if configuration does not specify one.

Returns
string One of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 103 of file JobQueueRedis.php.

◆ popAndAcquireBlob()

JobQueueRedis::popAndAcquireBlob ( RedisConnRef  $conn)
protected
Parameters
RedisConnRef$conn
Returns
array serialized string or false
Exceptions
RedisException

Definition at line 384 of file JobQueueRedis.php.

References are, array(), as, getQueueKey(), RedisConnRef\luaEval(), of, off, that, and then.

Referenced by doPop().

◆ popAndDeleteBlob()

JobQueueRedis::popAndDeleteBlob ( RedisConnRef  $conn)
protected
Parameters
RedisConnRef$conn
Returns
array serialized string or false
Exceptions
RedisException

Definition at line 351 of file JobQueueRedis.php.

References are, array(), data, getQueueKey(), RedisConnRef\luaEval(), of, off, that, and then.

Referenced by doPop().

◆ pushBlobs()

JobQueueRedis::pushBlobs ( RedisConnRef  $conn,
array  $items 
)
protected
Parameters
RedisConnRef$conn
array$itemsList of results from JobQueueRedis::getNewJobFields()
Returns
int Number of jobs inserted (duplicates are ignored)
Exceptions
RedisException

Definition at line 251 of file JobQueueRedis.php.

References $args, are, array(), as, getQueueKey(), RedisConnRef\luaEval(), of, release, serialize(), that, and then.

Referenced by doBatchPush().

◆ recyclePruneAndUndelayJobs()

JobQueueRedis::recyclePruneAndUndelayJobs ( )

Recycle or destroy any jobs that have been claimed for too long and release any ready delayed jobs into the queue.

Returns
int Number of jobs recycled/deleted/undelayed
Exceptions
MWException|JobQueueError

Definition at line 644 of file JobQueueRedis.php.

References $count, $e, $res, are, array(), as, from, getConnection(), getQueueKey(), in, JobQueue\incrStats(), is(), list, of, on, that, then, throwRedisException(), too, type, and wiki.

Referenced by doPop().

◆ serialize()

JobQueueRedis::serialize ( array  $fields)
protected
Parameters
array$fields
Returns
string Serialized and possibly compressed version of $fields

Definition at line 790 of file JobQueueRedis.php.

References $blob, array(), and object.

Referenced by getNewJobFields(), and pushBlobs().

◆ setTestingPrefix()

JobQueueRedis::setTestingPrefix (   $key)
Parameters
$keystring
Returns
void

Reimplemented from JobQueue.

Definition at line 866 of file JobQueueRedis.php.

References $key, and key.

◆ supportedOrders()

JobQueueRedis::supportedOrders ( )
protected

Get the allowed queue orders for configuration validation.

Returns
array Subset of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 99 of file JobQueueRedis.php.

References array().

◆ supportsDelayedJobs()

JobQueueRedis::supportsDelayedJobs ( )
protected

Find out if delayed jobs are supported for configuration validation.

Returns
bool Whether delayed jobs are supported

Reimplemented from JobQueue.

Definition at line 107 of file JobQueueRedis.php.

◆ throwRedisException()

◆ unserialize()

JobQueueRedis::unserialize (   $blob)
protected
Parameters
string$blob
Returns
array|bool Unserialized version of $blob or false

Definition at line 809 of file JobQueueRedis.php.

References $blob.

Referenced by doPop(), and getJobFromUidInternal().

Member Data Documentation

◆ $compression

string JobQueueRedis::$compression
protected

Compression method to use *.

Definition at line 64 of file JobQueueRedis.php.

◆ $key

string JobQueueRedis::$key
protected

Key to prefix the queue keys with (used for testing) *.

Definition at line 68 of file JobQueueRedis.php.

Referenced by doDeduplicateRootJob(), and setTestingPrefix().

◆ $maximumPeriodicTaskSeconds

null int JobQueueRedis::$maximumPeriodicTaskSeconds
protected

maximum seconds between execution of periodic tasks.

Used to speed up testing but should otherwise be left unset.

Definition at line 73 of file JobQueueRedis.php.

◆ $redisPool

RedisConnectionPool JobQueueRedis::$redisPool
protected

Definition at line 60 of file JobQueueRedis.php.

◆ $server

string JobQueueRedis::$server
protected

Server address *.

Definition at line 62 of file JobQueueRedis.php.

Referenced by getCoalesceLocationInternal().

◆ MAX_AGE_PRUNE

const JobQueueRedis::MAX_AGE_PRUNE = 604800

Definition at line 66 of file JobQueueRedis.php.


The documentation for this class was generated from the following file: