Redis-backed job queue storage. More...

Collaboration diagram for JobQueueRedis:

[legend]

Public Member Functions
	__construct (array $params)

	getAllAbandonedJobs ()

	getAllAcquiredJobs ()

	getAllDelayedJobs ()

	getAllQueuedJobs ()

	getCoalesceLocationInternal ()
	Do not use this function outside of JobQueue/JobQueueGroup.

	getJobFromUidInternal ( $uid, $conn)
	This function should not be called outside JobQueueRedis.

	getServerQueuesWithJobs ()

Public Member Functions inherited from JobQueue
	ack (RunnableJob $job)
	Acknowledge that a job was completed.

	batchPush (array $jobs, $flags=0)
	Push a batch of jobs into the queue.

	deduplicateRootJob (IJobSpecification $job)
	Register the "root job" of a given job into the queue for de-duplication.

	delayedJobsEnabled ()

	delete ()
	Delete all unclaimed and delayed jobs from the queue.

	flushCaches ()
	Clear any process and persistent caches.

	getAbandonedCount ()
	Get the number of acquired jobs that can no longer be attempted.

	getAcquiredCount ()
	Get the number of acquired jobs (these are temporarily out of the queue).

	getDelayedCount ()
	Get the number of delayed jobs (these are temporarily out of the queue).

	getDomain ()

	getOrder ()

	getReadOnlyReason ()

	getSiblingQueueSizes (array $types)
	Check the size of each of the given queues.

	getSiblingQueuesWithJobs (array $types)
	Check whether each of the given queues are empty.

	getSize ()
	Get the number of available (unacquired, non-delayed) jobs in the queue.

	getType ()

	isEmpty ()
	Quickly check if the queue has no available (unacquired, non-delayed) jobs.

	pop ()
	Pop a job off of the queue.

	push ( $jobs, $flags=0)
	Push one or more jobs into the queue.

	waitForBackups ()
	Wait for any replica DBs or backup servers to catch up.

Protected Member Functions
	doAck (RunnableJob $job)

	doBatchPush (array $jobs, $flags)

	doDeduplicateRootJob (IJobSpecification $job)

	doDelete ()

	doGetAbandonedCount ()

	doGetAcquiredCount ()

	doGetDelayedCount ()

	doGetSiblingQueueSizes (array $types)

	doGetSiblingQueuesWithJobs (array $types)

	doGetSize ()

	doIsEmpty ()

	doIsRootJobOldDuplicate (IJobSpecification $job)

	doPop ()

	getConnection ()
	Get a connection to the server that handles all sub-queues for this queue.

	getJobFromFields (array $fields)

	getJobIterator (RedisConnRef $conn, array $uids)

	getNewJobFields (IJobSpecification $job)

	handleErrorAndMakeException (RedisConnRef $conn, $e)

	optimalOrder ()
	Get the default queue order to use if configuration does not specify one.

	popAndAcquireBlob (RedisConnRef $conn)

	pushBlobs (RedisConnRef $conn, array $items)

	serialize (array $fields)

	supportedOrders ()
	Get the allowed queue orders for configuration validation.

	supportsDelayedJobs ()
	Find out if delayed jobs are supported for configuration validation.

	unserialize ( $blob)

Protected Member Functions inherited from JobQueue
	assertNotReadOnly ()

	doFlushCaches ()

	doWaitForBackups ()

	factoryJob ( $command, $params)

	getRootJobCacheKey ( $signature, $type)

	incrStats ( $key, $type, $delta=1)
	Call StatsdDataFactoryInterface::updateCount() for the queue overall and for the queue type.

	isRootJobOldDuplicate (IJobSpecification $job)
	Check if the "root" job of a given job has been superseded by a newer one.

	supportsTypeAgnostic ()
	Subclasses should set this to true if they support type agnostic queues.

Protected Attributes
string	$compression
	Compression method to use.

LoggerInterface	$logger

RedisConnectionPool	$redisPool

string	$server
	Server address.

Protected Attributes inherited from JobQueue
int	$claimTTL
	Time to live in seconds.

string	$domain
	DB domain ID.

GlobalIdGenerator	$idGenerator

int	$maxTries
	Maximum number of times to try a job.

string	$order
	Job priority for pop()

string false	$readOnlyReason
	Read only rationale (or false if r/w)

StatsdDataFactoryInterface	$stats

string	$type
	Job type.

bool	$typeAgnostic

WANObjectCache	$wanCache

const	QOS_ATOMIC = 1

const	ROOTJOB_TTL = 28 * 24 * 3600

Additional Inherited Members
Static Public Member Functions inherited from JobQueue
static	factory (array $params)
	Get a job queue object of the specified type.

Detailed Description

Redis-backed job queue storage.

This is a faster and less resource-intensive job queue than JobQueueDB. All data for a queue using this class is placed into one redis server.

When used on a wiki farm, you can optionally use the redisJobRunnerService background service from the mediawiki/services/jobrunner.git repository, to run jobs from a central system rather than per-wiki via one of the default job runners (e.g. maintenance/runJobs.php).

There are eight main redis keys (per queue) used to track jobs:

l-unclaimed : A list of job IDs used for ready unclaimed jobs
z-claimed : A sorted set of (job ID, UNIX timestamp as score) used for job retries
z-abandoned : A sorted set of (job ID, UNIX timestamp as score) used for broken jobs
z-delayed : A sorted set of (job ID, UNIX timestamp as score) used for delayed jobs
h-idBySha1 : A hash of (SHA1 => job ID) for unclaimed jobs used for de-duplication
h-sha1ById : A hash of (job ID => SHA1) for unclaimed jobs used for de-duplication
h-attempts : A hash of (job ID => attempt count) used for job claiming/retries
h-data : A hash of (job ID => serialized blobs) for job storage

A job ID can be in only one of z-delayed, l-unclaimed, z-claimed, and z-abandoned. If an ID appears in any of those lists, it should have a h-data entry for its ID. If a job has a SHA1 de-duplication value and its ID is in l-unclaimed or z-delayed, then there should be no other such jobs with that SHA1. Every h-idBySha1 entry has an h-sha1ById entry and every h-sha1ById must refer to an ID that is l-unclaimed. If a job has its ID in z-claimed or z-abandoned, then it must also have an h-attempts entry for its ID.

The following keys are used to track queue states:

s-queuesWithJobs : A set of all queues with non-abandoned jobs

The background service takes care of undelaying, recycling, and pruning jobs as well as removing s-queuesWithJobs entries as queues empty.

Additionally, "rootjob:* keys track "root jobs" used for additional de-duplication. Aside from root job keys, all keys have no expiry, and are only removed when jobs are run. All the keys are prefixed with the relevant wiki ID information.

This class requires Redis 2.6 or later as it uses Lua scripting for fast atomic operations. Additionally, it should be noted that redis has different persistence modes, such as rdb snapshots, journaling, or no persistence. Appropriate configuration should be made on the servers based on what queues are using it and what tolerance they have.

Since: 1.22

Definition at line 73 of file JobQueueRedis.php.

Constructor & Destructor Documentation

◆ __construct()

JobQueueRedis::__construct ( array $params )

Parameters

array

$params

Possible keys:

redisConfig : An array of parameters to RedisConnectionPool::__construct(). Note that the serializer option is ignored as "none" is always used.
redisServer : A hostname/port combination or the absolute path of a UNIX socket. If a hostname is specified but no port, the standard port number 6379 will be used. Required.
compression : The type of compression to use; one of (none,gzip).
daemonized : Set to true if the redisJobRunnerService runs in the background. This will disable job recycling/undelaying from the MediaWiki side to avoid redundancy and out-of-sync configuration.

Reimplemented from JobQueue.

Definition at line 98 of file JobQueueRedis.php.

References $params.

Member Function Documentation

◆ doAck()

JobQueueRedis::doAck ( RunnableJob $job )

protected

See also: JobQueue::doAck()

Parameters

RunnableJob $job

Returns: RunnableJob|bool

Exceptions

UnexpectedValueException
JobQueueError

Reimplemented from JobQueue.

Definition at line 393 of file JobQueueRedis.php.

References $job, getConnection(), handleErrorAndMakeException(), JobQueue\incrStats(), and wfDebugLog().

◆ doBatchPush()

JobQueueRedis::doBatchPush	(	array	$jobs,
			$flags )

protected

See also: JobQueue::doBatchPush()

Parameters

IJobSpecification[]	$jobs
int	$flags

Returns: void

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 200 of file JobQueueRedis.php.

References $job, getConnection(), getNewJobFields(), handleErrorAndMakeException(), JobQueue\incrStats(), pushBlobs(), and wfDebugLog().

◆ doDeduplicateRootJob()

JobQueueRedis::doDeduplicateRootJob ( IJobSpecification $job )

protected

See also: JobQueue::doDeduplicateRootJob()

Parameters

IJobSpecification $job

Returns: bool

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 447 of file JobQueueRedis.php.

References $job, $params, getConnection(), JobQueue\getRootJobCacheKey(), IJobSpecification\getType(), and handleErrorAndMakeException().

◆ doDelete()

JobQueueRedis::doDelete ( )

protected

See also: JobQueue::doDelete()

Returns: bool

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 498 of file JobQueueRedis.php.

References getConnection(), and handleErrorAndMakeException().

◆ doGetAbandonedCount()

JobQueueRedis::doGetAbandonedCount ( )

protected

See also: JobQueue::doGetAbandonedCount()

Returns: int

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 184 of file JobQueueRedis.php.

References getConnection(), and handleErrorAndMakeException().

◆ doGetAcquiredCount()

JobQueueRedis::doGetAcquiredCount ( )

protected

See also: JobQueue::doGetAcquiredCount()

Returns: int

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 152 of file JobQueueRedis.php.

References getConnection(), and handleErrorAndMakeException().

◆ doGetDelayedCount()

JobQueueRedis::doGetDelayedCount ( )

protected

See also: JobQueue::doGetDelayedCount()

Returns: int

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 170 of file JobQueueRedis.php.

References getConnection(), and handleErrorAndMakeException().

◆ doGetSiblingQueueSizes()

JobQueueRedis::doGetSiblingQueueSizes ( array $types )

protected

Stability: stable: to override

See also: JobQueue::getSiblingQueuesSize()

Parameters

array $types List of queues types

Returns: array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 607 of file JobQueueRedis.php.

References JobQueue\$type, getConnection(), and handleErrorAndMakeException().

Referenced by doGetSiblingQueuesWithJobs().

◆ doGetSiblingQueuesWithJobs()

JobQueueRedis::doGetSiblingQueuesWithJobs ( array $types )

protected

Stability: stable: to override

See also: JobQueue::getSiblingQueuesWithJobs()

Parameters

array $types List of queues types

Returns: array|null (list of queue types) or null if unsupported

Reimplemented from JobQueue.

Definition at line 603 of file JobQueueRedis.php.

References doGetSiblingQueueSizes().

◆ doGetSize()

JobQueueRedis::doGetSize ( )

protected

See also: JobQueue::doGetSize()

Returns: int

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 138 of file JobQueueRedis.php.

References getConnection(), and handleErrorAndMakeException().

Referenced by doIsEmpty().

◆ doIsEmpty()

JobQueueRedis::doIsEmpty ( )

protected

See also: JobQueue::doIsEmpty()

Returns: bool

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 129 of file JobQueueRedis.php.

References doGetSize().

◆ doIsRootJobOldDuplicate()

JobQueueRedis::doIsRootJobOldDuplicate ( IJobSpecification $job )

protected

See also: JobQueue::doIsRootJobOldDuplicate()

Parameters

IJobSpecification $job

Returns: bool

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 475 of file JobQueueRedis.php.

References $job, $params, getConnection(), JobQueue\getRootJobCacheKey(), IJobSpecification\getType(), and handleErrorAndMakeException().

◆ doPop()

JobQueueRedis::doPop ( )

protected

See also: JobQueue::doPop()

Returns: RunnableJob|false

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 319 of file JobQueueRedis.php.

References $job, getConnection(), getJobFromFields(), handleErrorAndMakeException(), JobQueue\incrStats(), popAndAcquireBlob(), unserialize(), and wfDebugLog().

◆ getAllAbandonedJobs()

JobQueueRedis::getAllAbandonedJobs ( )

See also: JobQueue::getAllAbandonedJobs()

Returns: Iterator<RunnableJob>

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 571 of file JobQueueRedis.php.

References getConnection(), getJobIterator(), and handleErrorAndMakeException().

◆ getAllAcquiredJobs()

JobQueueRedis::getAllAcquiredJobs ( )

See also: JobQueue::getAllAcquiredJobs()

Returns: Iterator<RunnableJob>

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 555 of file JobQueueRedis.php.

References getConnection(), getJobIterator(), and handleErrorAndMakeException().

◆ getAllDelayedJobs()

JobQueueRedis::getAllDelayedJobs ( )

See also: JobQueue::getAllDelayedJobs()

Returns: Iterator<RunnableJob>

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 539 of file JobQueueRedis.php.

References getConnection(), getJobIterator(), and handleErrorAndMakeException().

◆ getAllQueuedJobs()

JobQueueRedis::getAllQueuedJobs ( )

See also: JobQueue::getAllQueuedJobs()

Returns: Iterator<RunnableJob>

Exceptions

JobQueueError

Reimplemented from JobQueue.

Definition at line 523 of file JobQueueRedis.php.

References getConnection(), getJobIterator(), and handleErrorAndMakeException().

◆ getCoalesceLocationInternal()

JobQueueRedis::getCoalesceLocationInternal ( )

Do not use this function outside of JobQueue/JobQueueGroup.

Stability: stable: to override

Returns: string|null

Since: 1.22

Reimplemented from JobQueue.

Definition at line 599 of file JobQueueRedis.php.

References $server.

◆ getConnection()

JobQueueRedis::getConnection ( )

protected

Get a connection to the server that handles all sub-queues for this queue.

Returns: RedisConnRef|Redis

Exceptions

JobQueueConnectionError

Definition at line 764 of file JobQueueRedis.php.

Referenced by doAck(), doBatchPush(), doDeduplicateRootJob(), doDelete(), doGetAbandonedCount(), doGetAcquiredCount(), doGetDelayedCount(), doGetSiblingQueueSizes(), doGetSize(), doIsRootJobOldDuplicate(), doPop(), getAllAbandonedJobs(), getAllAcquiredJobs(), getAllDelayedJobs(), getAllQueuedJobs(), and getServerQueuesWithJobs().

◆ getJobFromFields()

JobQueueRedis::getJobFromFields ( array $fields )

protected

Parameters

array $fields

Returns: RunnableJob|false

Definition at line 711 of file JobQueueRedis.php.

References $job, $params, and JobQueue\factoryJob().

Referenced by doPop().

◆ getJobFromUidInternal()

JobQueueRedis::getJobFromUidInternal	(		$uid,
			$conn )

This function should not be called outside JobQueueRedis.

Parameters

string	$uid
RedisConnRef \| Redis	$conn

Returns: RunnableJob|false Returns false if the job does not exist

Exceptions

JobQueueError
UnexpectedValueException

Definition at line 638 of file JobQueueRedis.php.

References $job, $params, JobQueue\factoryJob(), handleErrorAndMakeException(), and unserialize().

Referenced by getJobIterator().

◆ getJobIterator()

JobQueueRedis::getJobIterator	(	RedisConnRef	$conn,
		array	$uids )

protected

Parameters

RedisConnRef	$conn
array	$uids	List of job UUIDs

Returns: MappedIterator<RunnableJob>

Definition at line 587 of file JobQueueRedis.php.

References $job, and getJobFromUidInternal().

Referenced by getAllAbandonedJobs(), getAllAcquiredJobs(), getAllDelayedJobs(), and getAllQueuedJobs().

◆ getNewJobFields()

JobQueueRedis::getNewJobFields ( IJobSpecification $job )

protected

Parameters

IJobSpecification $job

Returns: array

Definition at line 689 of file JobQueueRedis.php.

References $job, NS_SPECIAL, and serialize().

Referenced by doBatchPush().

◆ getServerQueuesWithJobs()

JobQueueRedis::getServerQueuesWithJobs ( )

Returns: array List of (wiki,type) tuples for queues with non-abandoned jobs

Exceptions

JobQueueConnectionError
JobQueueError

Definition at line 669 of file JobQueueRedis.php.

References getConnection(), and handleErrorAndMakeException().

◆ handleErrorAndMakeException()

JobQueueRedis::handleErrorAndMakeException	(	RedisConnRef	$conn,
			$e )

protected

Parameters

RedisConnRef	$conn
RedisException	$e

Returns: JobQueueError

Definition at line 779 of file JobQueueRedis.php.

Referenced by doAck(), doBatchPush(), doDeduplicateRootJob(), doDelete(), doGetAbandonedCount(), doGetAcquiredCount(), doGetDelayedCount(), doGetSiblingQueueSizes(), doGetSize(), doIsRootJobOldDuplicate(), doPop(), getAllAbandonedJobs(), getAllAcquiredJobs(), getAllDelayedJobs(), getAllQueuedJobs(), getJobFromUidInternal(), and getServerQueuesWithJobs().

◆ optimalOrder()

JobQueueRedis::optimalOrder ( )

protected

Get the default queue order to use if configuration does not specify one.

Returns: string One of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 116 of file JobQueueRedis.php.

◆ popAndAcquireBlob()

JobQueueRedis::popAndAcquireBlob ( RedisConnRef $conn )

protected

Parameters

RedisConnRef $conn

Returns: array Serialized string or false

Exceptions

RedisException

Definition at line 352 of file JobQueueRedis.php.

References Wikimedia\ObjectCache\RedisConnRef\luaEval().

Referenced by doPop().

◆ pushBlobs()

JobQueueRedis::pushBlobs	(	RedisConnRef	$conn,
		array	$items )

protected

Parameters

RedisConnRef	$conn
array[]	$items	List of results from JobQueueRedis::getNewJobFields()

Returns: int Number of jobs inserted (duplicates are ignored)

Exceptions

RedisException

Definition at line 254 of file JobQueueRedis.php.

References Wikimedia\ObjectCache\RedisConnRef\luaEval(), and serialize().

Referenced by doBatchPush().

◆ serialize()

JobQueueRedis::serialize ( array $fields )

protected

Parameters

array $fields

Returns: string Serialized and possibly compressed version of $fields

Definition at line 726 of file JobQueueRedis.php.

References serialize().

Referenced by getNewJobFields(), pushBlobs(), and serialize().

◆ supportedOrders()

JobQueueRedis::supportedOrders ( )

protected

Get the allowed queue orders for configuration validation.

Returns: array Subset of (random, timestamp, fifo, undefined)

Reimplemented from JobQueue.

Definition at line 112 of file JobQueueRedis.php.

◆ supportsDelayedJobs()

JobQueueRedis::supportsDelayedJobs ( )

protected

Find out if delayed jobs are supported for configuration validation.

Stability: stable: to override

Returns: bool Whether delayed jobs are supported

Reimplemented from JobQueue.

Definition at line 120 of file JobQueueRedis.php.

◆ unserialize()

JobQueueRedis::unserialize ( $blob )

protected

Parameters

string $blob

Returns: array|false Unserialized version of $blob or false

Definition at line 745 of file JobQueueRedis.php.

References unserialize().

Referenced by doPop(), getJobFromUidInternal(), and unserialize().

Member Data Documentation

◆ $compression

string JobQueueRedis::$compression

protected

Compression method to use.

Definition at line 82 of file JobQueueRedis.php.

◆ $logger

LoggerInterface JobQueueRedis::$logger

protected

Definition at line 77 of file JobQueueRedis.php.

◆ $redisPool

RedisConnectionPool JobQueueRedis::$redisPool

protected

Definition at line 75 of file JobQueueRedis.php.

◆ $server

string JobQueueRedis::$server

protected

Server address.

Definition at line 80 of file JobQueueRedis.php.

Referenced by getCoalesceLocationInternal().

The documentation for this class was generated from the following file:

includes/jobqueue/JobQueueRedis.php

Public Member Functions

Protected Member Functions

Protected Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ doAck()

◆ doBatchPush()

◆ doDeduplicateRootJob()

◆ doDelete()

◆ doGetAbandonedCount()

◆ doGetAcquiredCount()

◆ doGetDelayedCount()

◆ doGetSiblingQueueSizes()

◆ doGetSiblingQueuesWithJobs()

◆ doGetSize()

◆ doIsEmpty()

◆ doIsRootJobOldDuplicate()

◆ doPop()

◆ getAllAbandonedJobs()

◆ getAllAcquiredJobs()

◆ getAllDelayedJobs()

◆ getAllQueuedJobs()

◆ getCoalesceLocationInternal()

◆ getConnection()

◆ getJobFromFields()

◆ getJobFromUidInternal()

◆ getJobIterator()

◆ getNewJobFields()

◆ getServerQueuesWithJobs()

◆ handleErrorAndMakeException()

◆ optimalOrder()

◆ popAndAcquireBlob()

◆ pushBlobs()

◆ serialize()

◆ supportedOrders()

◆ supportsDelayedJobs()

◆ unserialize()

Member Data Documentation

◆ $compression

◆ $logger

◆ $redisPool

◆ $server