|
MediaWiki master
|
Redis-backed job queue storage. More...
Inherits MediaWiki\JobQueue\JobQueue.

Public Member Functions | |
| __construct (array $params) | |
| getAllAbandonedJobs () | |
| getAllAcquiredJobs () | |
| getAllDelayedJobs () | |
| getAllQueuedJobs () | |
| getCoalesceLocationInternal () | |
Do not use this function outside of JobQueue/JobQueueGroup.
| |
| getJobFromUidInternal ( $uid, $conn) | |
| This function should not be called outside JobQueueRedis. | |
| getServerQueuesWithJobs () | |
Public Member Functions inherited from MediaWiki\JobQueue\JobQueue | |
| ack (RunnableJob $job) | |
| Acknowledge that a job was completed. | |
| batchPush (array $jobs, $flags=0) | |
| Push a batch of jobs into the queue. | |
| deduplicateRootJob (IJobSpecification $job) | |
| Register the "root job" of a given job into the queue for de-duplication. | |
| delayedJobsEnabled () | |
| delete () | |
| Delete all unclaimed and delayed jobs from the queue. | |
| flushCaches () | |
| Clear any process and persistent caches. | |
| getAbandonedCount () | |
| Get the number of acquired jobs that can no longer be attempted. | |
| getAcquiredCount () | |
| Get the number of acquired jobs (these are temporarily out of the queue). | |
| getDelayedCount () | |
| Get the number of delayed jobs (these are temporarily out of the queue). | |
| getDomain () | |
| getOrder () | |
| getReadOnlyReason () | |
| getSiblingQueueSizes (array $types) | |
| Check the size of each of the given queues. | |
| getSiblingQueuesWithJobs (array $types) | |
| Check whether each of the given queues are empty. | |
| getSize () | |
| Get the number of available (unacquired, non-delayed) jobs in the queue. | |
| getType () | |
| isEmpty () | |
| Quickly check if the queue has no available (unacquired, non-delayed) jobs. | |
| pop () | |
| Pop a job off of the queue. | |
| push ( $jobs, $flags=0) | |
| Push one or more jobs into the queue. | |
| waitForBackups () | |
| Wait for any replica DBs or backup servers to catch up. | |
Protected Member Functions | ||||
| doAck (RunnableJob $job) | ||||
| doBatchPush (array $jobs, $flags) | ||||
| doDeduplicateRootJob (IJobSpecification $job) | ||||
| doDelete () | ||||
| doGetAbandonedCount () | ||||
| doGetAcquiredCount () | ||||
| doGetDelayedCount () | ||||
| doGetSiblingQueueSizes (array $types) | ||||
| ||||
| doGetSiblingQueuesWithJobs (array $types) | ||||
| ||||
| doGetSize () | ||||
| doIsEmpty () | ||||
| doIsRootJobOldDuplicate (IJobSpecification $job) | ||||
| doPop () | ||||
| getConnection () | ||||
| Get a connection to the server that handles all sub-queues for this queue. | ||||
| getJobFromFields (array $fields) | ||||
| getJobIterator (RedisConnRef $conn, array $uids) | ||||
| getNewJobFields (IJobSpecification $job) | ||||
| handleErrorAndMakeException (RedisConnRef $conn, $e) | ||||
| optimalOrder () | ||||
Get the default queue order to use if configuration does not specify one.
| ||||
| popAndAcquireBlob (RedisConnRef $conn) | ||||
| pushBlobs (RedisConnRef $conn, array $items) | ||||
| serialize (array $fields) | ||||
| supportedOrders () | ||||
Get the allowed queue orders for configuration validation.
| ||||
| supportsDelayedJobs () | ||||
Find out if delayed jobs are supported for configuration validation.
| ||||
| unserialize ( $blob) | ||||
Protected Member Functions inherited from MediaWiki\JobQueue\JobQueue | ||||
| assertNotReadOnly () | ||||
| doFlushCaches () | ||||
| doWaitForBackups () | ||||
| factoryJob ( $command, $params) | ||||
| getRootJobCacheKey ( $signature, $type) | ||||
| incrStats ( $event, $type, $delta=1) | ||||
| Call StatsFactory::incrementBy() for the queue overall and for the queue type. | ||||
| isRootJobOldDuplicate (IJobSpecification $job) | ||||
| Check if the "root" job of a given job has been superseded by a newer one. | ||||
| supportsTypeAgnostic () | ||||
| Subclasses should set this to true if they support type agnostic queues. | ||||
Protected Attributes | |
| string | $compression |
| Compression method to use. | |
| LoggerInterface | $logger |
| RedisConnectionPool | $redisPool |
| string | $server |
| Server address. | |
Protected Attributes inherited from MediaWiki\JobQueue\JobQueue | |
| int | $claimTTL |
| Time to live in seconds. | |
| string | $domain |
| DB domain ID. | |
| GlobalIdGenerator | $idGenerator |
| int | $maxTries |
| Maximum number of times to try a job. | |
| string | $order |
| Job priority for pop() | |
| string false | $readOnlyReason |
| Read only rationale (or false if r/w) | |
| StatsFactory | $stats |
| string | $type |
| Job type. | |
| bool | $typeAgnostic |
| WANObjectCache | $wanCache |
| const | QOS_ATOMIC = 1 |
| const | ROOTJOB_TTL = 28 * 24 * 3600 |
Additional Inherited Members | |
Static Public Member Functions inherited from MediaWiki\JobQueue\JobQueue | |
| static | factory (array $params) |
| Get a job queue object of the specified type. | |
Redis-backed job queue storage.
This is a faster and less resource-intensive job queue than JobQueueDB. All data for a queue using this class is placed into one redis server.
When used on a wiki farm, you can optionally use the redisJobRunnerService background service from the mediawiki/services/jobrunner.git repository, to run jobs from a central system rather than per-wiki via one of the default job runners (e.g. maintenance/runJobs.php).
There are eight main redis keys (per queue) used to track jobs:
A job ID can be in only one of z-delayed, l-unclaimed, z-claimed, and z-abandoned. If an ID appears in any of those lists, it should have a h-data entry for its ID. If a job has a SHA1 de-duplication value and its ID is in l-unclaimed or z-delayed, then there should be no other such jobs with that SHA1. Every h-idBySha1 entry has an h-sha1ById entry and every h-sha1ById must refer to an ID that is l-unclaimed. If a job has its ID in z-claimed or z-abandoned, then it must also have an h-attempts entry for its ID.
The following keys are used to track queue states:
The background service takes care of undelaying, recycling, and pruning jobs as well as removing s-queuesWithJobs entries as queues empty.
Additionally, "rootjob:* keys track "root jobs" used for additional de-duplication. Aside from root job keys, all keys have no expiry, and are only removed when jobs are run. All the keys are prefixed with the relevant wiki ID information.
This class requires Redis 2.6 or later as it uses Lua scripting for fast atomic operations. Additionally, it should be noted that redis has different persistence modes, such as rdb snapshots, journaling, or no persistence. Appropriate configuration should be made on the servers based on what queues are using it and what tolerance they have.
Definition at line 69 of file JobQueueRedis.php.
| MediaWiki\JobQueue\JobQueueRedis::__construct | ( | array | $params | ) |
| array | $params | Possible keys:
|
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 94 of file JobQueueRedis.php.
References Wikimedia\ObjectCache\RedisConnectionPool\singleton().
|
protected |
| RunnableJob | $job |
| UnexpectedValueException | |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 392 of file JobQueueRedis.php.
References $job, MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException(), MediaWiki\JobQueue\JobQueue\incrStats(), and wfDebugLog().
|
protected |
| IJobSpecification[] | $jobs | |
| int | $flags |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 199 of file JobQueueRedis.php.
References $job, MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueueRedis\getNewJobFields(), MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException(), MediaWiki\JobQueue\JobQueue\incrStats(), MediaWiki\JobQueue\JobQueueRedis\pushBlobs(), and wfDebugLog().
|
protected |
| IJobSpecification | $job |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 446 of file JobQueueRedis.php.
References $job, MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueue\getRootJobCacheKey(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
|
protected |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 497 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
|
protected |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 183 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
|
protected |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 151 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
|
protected |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 169 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
|
protected |
| array | $types | List of queues types |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 609 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueue\$type, MediaWiki\JobQueue\JobQueueRedis\getConnection(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
Referenced by MediaWiki\JobQueue\JobQueueRedis\doGetSiblingQueuesWithJobs().
|
protected |
| array | $types | List of queues types |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 604 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\doGetSiblingQueueSizes().
|
protected |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 137 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
Referenced by MediaWiki\JobQueue\JobQueueRedis\doIsEmpty().
|
protected |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 128 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\doGetSize().
|
protected |
| IJobSpecification | $job |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 474 of file JobQueueRedis.php.
References $job, MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueue\getRootJobCacheKey(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
|
protected |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 318 of file JobQueueRedis.php.
References $job, MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueueRedis\getJobFromFields(), MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException(), MediaWiki\JobQueue\JobQueue\incrStats(), MediaWiki\JobQueue\JobQueueRedis\popAndAcquireBlob(), MediaWiki\JobQueue\JobQueueRedis\unserialize(), and wfDebugLog().
| MediaWiki\JobQueue\JobQueueRedis::getAllAbandonedJobs | ( | ) |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 570 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueueRedis\getJobIterator(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
| MediaWiki\JobQueue\JobQueueRedis::getAllAcquiredJobs | ( | ) |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 554 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueueRedis\getJobIterator(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
| MediaWiki\JobQueue\JobQueueRedis::getAllDelayedJobs | ( | ) |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 538 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueueRedis\getJobIterator(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
| MediaWiki\JobQueue\JobQueueRedis::getAllQueuedJobs | ( | ) |
| JobQueueError |
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 522 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), MediaWiki\JobQueue\JobQueueRedis\getJobIterator(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
| MediaWiki\JobQueue\JobQueueRedis::getCoalesceLocationInternal | ( | ) |
Do not use this function outside of JobQueue/JobQueueGroup.
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 599 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\$server.
|
protected |
Get a connection to the server that handles all sub-queues for this queue.
| JobQueueConnectionError |
Definition at line 766 of file JobQueueRedis.php.
Referenced by MediaWiki\JobQueue\JobQueueRedis\doAck(), MediaWiki\JobQueue\JobQueueRedis\doBatchPush(), MediaWiki\JobQueue\JobQueueRedis\doDeduplicateRootJob(), MediaWiki\JobQueue\JobQueueRedis\doDelete(), MediaWiki\JobQueue\JobQueueRedis\doGetAbandonedCount(), MediaWiki\JobQueue\JobQueueRedis\doGetAcquiredCount(), MediaWiki\JobQueue\JobQueueRedis\doGetDelayedCount(), MediaWiki\JobQueue\JobQueueRedis\doGetSiblingQueueSizes(), MediaWiki\JobQueue\JobQueueRedis\doGetSize(), MediaWiki\JobQueue\JobQueueRedis\doIsRootJobOldDuplicate(), MediaWiki\JobQueue\JobQueueRedis\doPop(), MediaWiki\JobQueue\JobQueueRedis\getAllAbandonedJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllAcquiredJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllDelayedJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllQueuedJobs(), and MediaWiki\JobQueue\JobQueueRedis\getServerQueuesWithJobs().
|
protected |
| array | $fields |
Definition at line 713 of file JobQueueRedis.php.
References $job, and MediaWiki\JobQueue\JobQueue\factoryJob().
Referenced by MediaWiki\JobQueue\JobQueueRedis\doPop().
| MediaWiki\JobQueue\JobQueueRedis::getJobFromUidInternal | ( | $uid, | |
| $conn ) |
This function should not be called outside JobQueueRedis.
| string | $uid | |
| RedisConnRef | Redis | $conn |
| JobQueueError | |
| UnexpectedValueException |
Definition at line 640 of file JobQueueRedis.php.
References $job, MediaWiki\JobQueue\JobQueue\factoryJob(), MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException(), and MediaWiki\JobQueue\JobQueueRedis\unserialize().
Referenced by MediaWiki\JobQueue\JobQueueRedis\getJobIterator().
|
protected |
| RedisConnRef | $conn | |
| array | $uids | List of job UUIDs |
Definition at line 586 of file JobQueueRedis.php.
References $job, and MediaWiki\JobQueue\JobQueueRedis\getJobFromUidInternal().
Referenced by MediaWiki\JobQueue\JobQueueRedis\getAllAbandonedJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllAcquiredJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllDelayedJobs(), and MediaWiki\JobQueue\JobQueueRedis\getAllQueuedJobs().
|
protected |
| IJobSpecification | $job |
Definition at line 691 of file JobQueueRedis.php.
References $job, NS_SPECIAL, and MediaWiki\JobQueue\JobQueueRedis\serialize().
Referenced by MediaWiki\JobQueue\JobQueueRedis\doBatchPush().
| MediaWiki\JobQueue\JobQueueRedis::getServerQueuesWithJobs | ( | ) |
| JobQueueConnectionError | |
| JobQueueError |
Definition at line 671 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\getConnection(), and MediaWiki\JobQueue\JobQueueRedis\handleErrorAndMakeException().
|
protected |
| RedisConnRef | $conn | |
| RedisException | $e |
Definition at line 781 of file JobQueueRedis.php.
Referenced by MediaWiki\JobQueue\JobQueueRedis\doAck(), MediaWiki\JobQueue\JobQueueRedis\doBatchPush(), MediaWiki\JobQueue\JobQueueRedis\doDeduplicateRootJob(), MediaWiki\JobQueue\JobQueueRedis\doDelete(), MediaWiki\JobQueue\JobQueueRedis\doGetAbandonedCount(), MediaWiki\JobQueue\JobQueueRedis\doGetAcquiredCount(), MediaWiki\JobQueue\JobQueueRedis\doGetDelayedCount(), MediaWiki\JobQueue\JobQueueRedis\doGetSiblingQueueSizes(), MediaWiki\JobQueue\JobQueueRedis\doGetSize(), MediaWiki\JobQueue\JobQueueRedis\doIsRootJobOldDuplicate(), MediaWiki\JobQueue\JobQueueRedis\doPop(), MediaWiki\JobQueue\JobQueueRedis\getAllAbandonedJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllAcquiredJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllDelayedJobs(), MediaWiki\JobQueue\JobQueueRedis\getAllQueuedJobs(), MediaWiki\JobQueue\JobQueueRedis\getJobFromUidInternal(), and MediaWiki\JobQueue\JobQueueRedis\getServerQueuesWithJobs().
|
protected |
Get the default queue order to use if configuration does not specify one.
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 114 of file JobQueueRedis.php.
|
protected |
| RedisConnRef | $conn |
| RedisException |
Definition at line 351 of file JobQueueRedis.php.
References Wikimedia\ObjectCache\RedisConnRef\luaEval().
Referenced by MediaWiki\JobQueue\JobQueueRedis\doPop().
|
protected |
| RedisConnRef | $conn | |
| array[] | $items | List of results from JobQueueRedis::getNewJobFields() |
| RedisException |
Definition at line 253 of file JobQueueRedis.php.
References Wikimedia\ObjectCache\RedisConnRef\luaEval(), and MediaWiki\JobQueue\JobQueueRedis\serialize().
Referenced by MediaWiki\JobQueue\JobQueueRedis\doBatchPush().
|
protected |
| array | $fields |
Definition at line 728 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\serialize().
Referenced by MediaWiki\JobQueue\JobQueueRedis\getNewJobFields(), MediaWiki\JobQueue\JobQueueRedis\pushBlobs(), and MediaWiki\JobQueue\JobQueueRedis\serialize().
|
protected |
Get the allowed queue orders for configuration validation.
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 109 of file JobQueueRedis.php.
|
protected |
Find out if delayed jobs are supported for configuration validation.
Reimplemented from MediaWiki\JobQueue\JobQueue.
Definition at line 119 of file JobQueueRedis.php.
|
protected |
| string | $blob |
Definition at line 747 of file JobQueueRedis.php.
References MediaWiki\JobQueue\JobQueueRedis\unserialize().
Referenced by MediaWiki\JobQueue\JobQueueRedis\doPop(), MediaWiki\JobQueue\JobQueueRedis\getJobFromUidInternal(), and MediaWiki\JobQueue\JobQueueRedis\unserialize().
|
protected |
Compression method to use.
Definition at line 78 of file JobQueueRedis.php.
|
protected |
Definition at line 73 of file JobQueueRedis.php.
|
protected |
Definition at line 71 of file JobQueueRedis.php.
|
protected |
Server address.
Definition at line 76 of file JobQueueRedis.php.
Referenced by MediaWiki\JobQueue\JobQueueRedis\getCoalesceLocationInternal().