MediaWiki REL1_32
|
Class to handle enqueueing and running of background jobs for federated queues. More...
Public Member Functions | |
getAllAbandonedJobs () | |
Get an iterator to traverse over all abandoned jobs in this queue. | |
getAllAcquiredJobs () | |
Get an iterator to traverse over all claimed jobs in this queue. | |
getAllDelayedJobs () | |
Get an iterator to traverse over all delayed jobs in this queue. | |
getAllQueuedJobs () | |
Get an iterator to traverse over all available jobs in this queue. | |
getCoalesceLocationInternal () | |
Do not use this function outside of JobQueue/JobQueueGroup. | |
Public Member Functions inherited from JobQueue | |
ack (Job $job) | |
Acknowledge that a job was completed. | |
batchPush (array $jobs, $flags=0) | |
Push a batch of jobs into the queue. | |
deduplicateRootJob (IJobSpecification $job) | |
Register the "root job" of a given job into the queue for de-duplication. | |
delayedJobsEnabled () | |
delete () | |
Deleted all unclaimed and delayed jobs from the queue. | |
flushCaches () | |
Clear any process and persistent caches. | |
getAbandonedCount () | |
Get the number of acquired jobs that can no longer be attempted. | |
getAcquiredCount () | |
Get the number of acquired jobs (these are temporarily out of the queue). | |
getDelayedCount () | |
Get the number of delayed jobs (these are temporarily out of the queue). | |
getOrder () | |
getReadOnlyReason () | |
getSiblingQueueSizes (array $types) | |
Check the size of each of the given queues. | |
getSiblingQueuesWithJobs (array $types) | |
Check whether each of the given queues are empty. | |
getSize () | |
Get the number of available (unacquired, non-delayed) jobs in the queue. | |
getType () | |
getWiki () | |
isEmpty () | |
Quickly check if the queue has no available (unacquired, non-delayed) jobs. | |
pop () | |
Pop a job off of the queue. | |
push ( $jobs, $flags=0) | |
Push one or more jobs into the queue. | |
waitForBackups () | |
Wait for any replica DBs or backup servers to catch up. | |
Protected Member Functions | |
__construct (array $params) | |
doAck (Job $job) | |
doBatchPush (array $jobs, $flags) | |
doDeduplicateRootJob (IJobSpecification $job) | |
doDelete () | |
doFlushCaches () | |
doGetAbandonedCount () | |
doGetAcquiredCount () | |
doGetDelayedCount () | |
doGetSiblingQueueSizes (array $types) | |
doGetSiblingQueuesWithJobs (array $types) | |
doGetSize () | |
doIsEmpty () | |
doIsRootJobOldDuplicate (Job $job) | |
doPop () | |
doWaitForBackups () | |
getCrossPartitionSum ( $type, $method) | |
logException (Exception $e) | |
optimalOrder () | |
Get the default queue order to use if configuration does not specify one. | |
supportedOrders () | |
Get the allowed queue orders for configuration validation. | |
supportsDelayedJobs () | |
Find out if delayed jobs are supported for configuration validation. | |
throwErrorIfAllPartitionsDown ( $down) | |
Throw an error if no partitions available. | |
tryJobInsertions (array $jobs, HashRing &$partitionRing, $flags) | |
Protected Member Functions inherited from JobQueue | |
assertNotReadOnly () | |
getRootJobCacheKey ( $signature) | |
isRootJobOldDuplicate (Job $job) | |
Check if the "root" job of a given job has been superseded by a newer one. | |
Protected Attributes | |
int | $maxPartitionsTry |
Maximum number of partitions to try. | |
JobQueue[] | $partitionQueues = [] |
(partition name => JobQueue) reverse sorted by weight | |
HashRing | $partitionRing |
Protected Attributes inherited from JobQueue | |
JobQueueAggregator | $aggr |
int | $claimTTL |
Time to live in seconds. | |
BagOStuff | $dupCache |
int | $maxTries |
Maximum number of times to try a job. | |
string | $order |
Job priority for pop() | |
string bool | $readOnlyReason |
Read only rationale (or false if r/w) | |
string | $type |
Job type. | |
string | $wiki |
Wiki ID. | |
Additional Inherited Members | |
Static Public Member Functions inherited from JobQueue | |
static | factory (array $params) |
Get a job queue object of the specified type. | |
static | incrStats ( $key, $type, $delta=1) |
Call wfIncrStats() for the queue overall and for the queue type. | |
Public Attributes inherited from JobQueue | |
const | QOS_ATOMIC = 1 |
const | ROOTJOB_TTL = 2419200 |
Class to handle enqueueing and running of background jobs for federated queues.
This class allows for queues to be partitioned into smaller queues. A partition is defined by the configuration for a JobQueue instance. For example, one can set $wgJobTypeConf['refreshLinks'] to point to a JobQueueFederated instance, which itself would consist of three JobQueueRedis instances, each using their own redis server. This would allow for the jobs to be split (evenly or based on weights) across multiple servers if a single server becomes impractical or expensive. Different JobQueue classes can be mixed.
The basic queue configuration (e.g. "order", "claimTTL") of a federated queue is inherited by the partition queues. Additional configuration defines what section each wiki is in, what partition queues each section uses (and their weight), and the JobQueue configuration for each partition. Some sections might only need a single queue partition, like the sections for groups of small wikis.
If used for performance, then $wgMainCacheType should be set to memcached/redis. Note that "fifo" cannot be used for the ordering, since the data is distributed. One can still use "timestamp" instead, as in "roughly timestamp ordered". Also, queue classes used by this should ignore down servers (with TTL) to avoid slowness.
Definition at line 48 of file JobQueueFederated.php.
|
protected |
array | $params | Possible keys:
|
MWException |
Reimplemented from JobQueue.
Definition at line 74 of file JobQueueFederated.php.
References $params, $section, JobQueue\$wiki, and JobQueue\factory().
|
protected |
Job | $job |
Reimplemented from JobQueue.
Definition at line 303 of file JobQueueFederated.php.
References $job.
|
protected |
IJobSpecification[] | $jobs | |
int | $flags |
Reimplemented from JobQueue.
Definition at line 177 of file JobQueueFederated.php.
References $e, $partitionRing, HashRing\getLiveLocationWeights(), and tryJobInsertions().
|
protected |
IJobSpecification | $job |
MWException |
Reimplemented from JobQueue.
Definition at line 326 of file JobQueueFederated.php.
|
protected |
MWException |
Reimplemented from JobQueue.
Definition at line 341 of file JobQueueFederated.php.
References $e, $queue, logException(), and throwErrorIfAllPartitionsDown().
|
protected |
Reimplemented from JobQueue.
Definition at line 370 of file JobQueueFederated.php.
References $queue.
|
protected |
Reimplemented from JobQueue.
Definition at line 152 of file JobQueueFederated.php.
References getCrossPartitionSum().
|
protected |
Reimplemented from JobQueue.
Definition at line 144 of file JobQueueFederated.php.
References getCrossPartitionSum().
|
protected |
Reimplemented from JobQueue.
Definition at line 148 of file JobQueueFederated.php.
References getCrossPartitionSum().
|
protected |
array | $types | List of queues types |
Reimplemented from JobQueue.
Definition at line 452 of file JobQueueFederated.php.
References $e, $queue, $result, JobQueue\$type, logException(), and throwErrorIfAllPartitionsDown().
|
protected |
array | $types | List of queues types |
Reimplemented from JobQueue.
Definition at line 426 of file JobQueueFederated.php.
References $e, $queue, logException(), and throwErrorIfAllPartitionsDown().
|
protected |
Reimplemented from JobQueue.
Definition at line 140 of file JobQueueFederated.php.
References getCrossPartitionSum().
|
protected |
Reimplemented from JobQueue.
Definition at line 124 of file JobQueueFederated.php.
References $e, $queue, logException(), and throwErrorIfAllPartitionsDown().
|
protected |
Job | $job |
Reimplemented from JobQueue.
Definition at line 311 of file JobQueueFederated.php.
|
protected |
Reimplemented from JobQueue.
Definition at line 271 of file JobQueueFederated.php.
References $e, $job, $queue, logException(), and throwErrorIfAllPartitionsDown().
|
protected |
Reimplemented from JobQueue.
Definition at line 356 of file JobQueueFederated.php.
References $e, $queue, logException(), and throwErrorIfAllPartitionsDown().
JobQueueFederated::getAllAbandonedJobs | ( | ) |
Get an iterator to traverse over all abandoned jobs in this queue.
JobQueueError |
Reimplemented from JobQueue.
Definition at line 410 of file JobQueueFederated.php.
References $queue.
JobQueueFederated::getAllAcquiredJobs | ( | ) |
Get an iterator to traverse over all claimed jobs in this queue.
Callers should be quick to iterator over it or few results will be returned due to jobs being acknowledged and deleted
JobQueueError |
Reimplemented from JobQueue.
Definition at line 399 of file JobQueueFederated.php.
References $queue.
JobQueueFederated::getAllDelayedJobs | ( | ) |
Get an iterator to traverse over all delayed jobs in this queue.
Note: results may be stale if the queue is concurrently modified.
JobQueueError |
Reimplemented from JobQueue.
Definition at line 388 of file JobQueueFederated.php.
References $queue.
JobQueueFederated::getAllQueuedJobs | ( | ) |
Get an iterator to traverse over all available jobs in this queue.
This does not include jobs that are currently acquired or delayed. Note: results may be stale if the queue is concurrently modified.
JobQueueError |
Reimplemented from JobQueue.
Definition at line 377 of file JobQueueFederated.php.
References $queue.
JobQueueFederated::getCoalesceLocationInternal | ( | ) |
Do not use this function outside of JobQueue/JobQueueGroup.
Reimplemented from JobQueue.
Definition at line 421 of file JobQueueFederated.php.
References serialize().
|
protected |
string | $type | |
string | $method |
Definition at line 161 of file JobQueueFederated.php.
References $e, $queue, logException(), and throwErrorIfAllPartitionsDown().
Referenced by doGetAbandonedCount(), doGetAcquiredCount(), doGetDelayedCount(), and doGetSize().
|
protected |
Definition at line 476 of file JobQueueFederated.php.
References $e, and wfDebugLog().
Referenced by doDelete(), doGetSiblingQueueSizes(), doGetSiblingQueuesWithJobs(), doIsEmpty(), doPop(), doWaitForBackups(), getCrossPartitionSum(), and tryJobInsertions().
|
protected |
Get the default queue order to use if configuration does not specify one.
Reimplemented from JobQueue.
Definition at line 110 of file JobQueueFederated.php.
|
protected |
Get the allowed queue orders for configuration validation.
Reimplemented from JobQueue.
Definition at line 105 of file JobQueueFederated.php.
|
protected |
Find out if delayed jobs are supported for configuration validation.
Reimplemented from JobQueue.
Definition at line 114 of file JobQueueFederated.php.
References $queue.
|
protected |
Throw an error if no partitions available.
int | $down | The number of up partitions down |
JobQueueError |
Definition at line 487 of file JobQueueFederated.php.
Referenced by doDelete(), doGetSiblingQueueSizes(), doGetSiblingQueuesWithJobs(), doIsEmpty(), doPop(), doWaitForBackups(), and getCrossPartitionSum().
array | $jobs | |
HashRing | &$partitionRing | |
int | $flags |
JobQueueError |
Definition at line 205 of file JobQueueFederated.php.
References $e, $job, $partitionRing, $queue, HashRing\ejectFromLiveRing(), HashRing\getLiveLocation(), HashRing\getLiveLocationWeights(), logException(), and serialize().
Referenced by doBatchPush().
|
protected |
Maximum number of partitions to try.
Definition at line 55 of file JobQueueFederated.php.
|
protected |
(partition name => JobQueue) reverse sorted by weight
Definition at line 52 of file JobQueueFederated.php.
|
protected |
Definition at line 50 of file JobQueueFederated.php.
Referenced by doBatchPush(), and tryJobInsertions().