RDBMS-based caching module. More...

Inheritance diagram for SqlBagOStuff:

[legend]

Collaboration diagram for SqlBagOStuff:

[legend]

Public Member Functions
	__construct ( $params)
	Create a new backend instance from configuration.

	createTables ()
	Create the shard tables on all databases (e.g.

	decr ( $key, $value=1, $flags=0)
	Decrease stored value of $key by $value while preserving its TTL.

	deleteAll ()
	Delete content of shard tables in every server.

	deleteObjectsExpiringBefore ( $timestamp, callable $progress=null, $limit=INF, string $tag=null)
	Delete all objects expiring before a certain date.

	doChangeTTLMulti (array $keys, $exptime, $flags=0)

	doLock ( $key, $timeout=6, $exptime=6)

	doUnlock ( $key)

	expireAll ()

	incr ( $key, $value=1, $flags=0)
	Increase stored value of $key by $value while preserving its TTL.

	incrWithInit ( $key, $exptime, $value=1, $init=null, $flags=0)
	Increase the value of the given key (no TTL change) if it exists or create it otherwise.

	makeKeyInternal ( $keyspace, $components)
	Make a cache key for the given keyspace and components.

Public Member Functions inherited from MediumSpecificBagOStuff
	__construct (array $params=[])

	add ( $key, $value, $exptime=0, $flags=0)
	Insert an item if it does not already exist.

	addBusyCallback (callable $workCallback)
	Let a callback be run to avoid wasting time on special blocking calls.

	changeTTL ( $key, $exptime=0, $flags=0)
	Change the expiration on a key if it exists.

	changeTTLMulti (array $keys, $exptime, $flags=0)
	Change the expiration of multiple keys that exist.

	clearLastError ()
	Clear the "last error" registry.

	delete ( $key, $flags=0)
	Delete an item.

	deleteMulti (array $keys, $flags=0)
	Batch deletion.

	get ( $key, $flags=0)
	Get an item with the given key.

	getLastError ()
	Get the "last error" registered; clearLastError() should be called manually.

	getMulti (array $keys, $flags=0)
	Get an associative array containing the item for each of the keys that have items.

	getQoS ( $flag)

	getSegmentationSize ()

	getSegmentedValueMaxSize ()

	lock ( $key, $timeout=6, $exptime=6, $rclass='')

	makeGlobalKey ( $collection,... $components)
	Make a cache key for the default keyspace and given components.

	makeKey ( $collection,... $components)
	Make a cache key for the global keyspace and given components.

	merge ( $key, callable $callback, $exptime=0, $attempts=10, $flags=0)
	Merge changes into the existing cache value (possibly creating a new one)

	set ( $key, $value, $exptime=0, $flags=0)
	Set an item.

	setMulti (array $valueByKey, $exptime=0, $flags=0)
	Batch insertion/replace.

	setNewPreparedValues (array $valueByKey)
	Make a "generic" reversible cache key from the given components.

	unlock ( $key)
	Release an advisory lock on a key string.

Public Member Functions inherited from BagOStuff
	getCurrentTime ()

	getLogger ()

	getScopedLock ( $key, $timeout=6, $expiry=30, $rclass='')
	Get a lightweight exclusive self-unlocking lock.

	getWithSetCallback ( $key, $exptime, $callback, $flags=0)
	Get an item with the given key, regenerating and setting it if not found.

	isKeyGlobal ( $key)
	Check whether a cache key is in the global keyspace.

	registerWrapperInfoForStats (string $prefixComponent, string $statsGroup, callable $collectionCallback)
	Register info about a caching layer class that uses BagOStuff as a backing store.

	setDebug ( $enabled)

	setLogger (LoggerInterface $logger)

	setMockTime (&$time)

Protected Member Functions
	doAdd ( $key, $value, $exptime=0, $flags=0)
	Insert an item if it does not already exist.

	doCas ( $casToken, $key, $value, $exptime=0, $flags=0)
	Check and set an item.

	doChangeTTL ( $key, $exptime, $flags)

	doDelete ( $key, $flags=0)
	Delete an item.

	doDeleteMulti (array $keys, $flags=0)

	doGet ( $key, $flags=0, &$casToken=null)

	doGetMulti (array $keys, $flags=0)
	Get an associative array containing the item for each of the keys that have items.

	doSet ( $key, $value, $exptime=0, $flags=0)
	Set an item.

	doSetMulti (array $data, $exptime=0, $flags=0)

	serialize ( $value)

	unserialize ( $value)

Protected Member Functions inherited from MediumSpecificBagOStuff
	cas ( $casToken, $key, $value, $exptime=0, $flags=0)
	Check and set an item.

	convertGenericKey ( $key)
	Convert a "generic" reversible cache key into one for this cache.

	debug ( $text)

	getExpirationAsTimestamp ( $exptime)
	Convert an optionally relative timestamp to an absolute time.

	getExpirationAsTTL ( $exptime)
	Convert an optionally absolute expiry time to a relative time.

	getSerialized ( $value, $key)
	Get the serialized form a value, using any applicable prepared value.

	guessSerialValueSize ( $value, $depth=0, &$loops=0)
	Estimate the size of a variable once serialized.

	isInteger ( $value)
	Check if a value is an integer.

	isRelativeExpiration ( $exptime)

	makeLockKey ( $key)

	makeValueOrSegmentList ( $key, $value, $exptime, $flags)
	Determine the entry (inline or segment list) to store under a key to save the value.

	mergeViaCas ( $key, callable $callback, $exptime, $attempts, $flags)

	resolveSegments ( $key, $mainValue)
	Get and reassemble the chunks of blob at the given key.

	setLastError ( $err)
	Set the "last error" registry.

	updateOpStats (string $op, array $keyInfo)

Protected Member Functions inherited from BagOStuff
	componentsFromGenericKey ( $key)
	Extract the components from a "generic" reversible cache key.

	determineKeyPrefixForStats ( $key)

	fieldHasFlags ( $field, $flags)

	genericKeyFromComponents (... $components)
	At a minimum, there must be a keyspace and collection name component.

	mergeFlagMaps (array $bags)
	Merge the flag maps of one or more BagOStuff objects into a "lowest common denominator" map.

	proxyCall ( $method, $arg0Sig, $resSig, array $genericArgs)
	Call a method on behalf of wrapper BagOStuff instance that uses "generic" keys.

Protected Attributes
Exception[]	$connFailureErrors = []
	Map of (shard index => Exception)

float[]	$connFailureTimes = []
	Map of (shard index => UNIX timestamps)

IMaintainableDatabase[]	$conns
	Map of (shard index => DB handle)

ILoadBalancer null	$globalKeyLb

int	$lastGarbageCollect = 0
	UNIX timestamp.

ILoadBalancer null	$localKeyLb

string null	$multiPrimaryModeType
	Multi-primary mode DB type ("mysql",...); null if not enabled.

int	$numTableShards = 1
	Number of table shards to use on each server.

int	$purgeLimit = 100
	Max expired rows to purge during randomized garbage collection.

int	$purgePeriod = 10
	Average number of writes required to trigger garbage collection.

bool	$replicaOnly
	Whether to use replicas instead of primaries (if using LoadBalancer)

array[]	$serverInfos = []
	(server index => server config)

string[]	$serverTags = []
	(server index => tag/host name)

string	$tableName = 'objectcache'

int	$writeBatchSize = 100

Protected Attributes inherited from MediumSpecificBagOStuff
callable[]	$busyCallbacks = []

int	$lastError = self::ERR_NONE
	ERR_* class constant.

array< string, array >	$locks = []
	Map of (key => (class, depth, expiry)

array[]	$preparedValues = []
	Map of (key => (PHP variable value, serialized value))

int	$segmentationSize
	Bytes; chunk size of segmented cache values.

int	$segmentedValueMaxSize
	Bytes; maximum total size of a segmented cache value.

int	$syncTimeout
	Seconds.

const	LOCK_DEPTH = 1

const	LOCK_EXPIRY = 3

const	LOCK_RCLASS = 0

const	LOCK_TIME = 2

const	METRIC_OP_ADD = 'add'

const	METRIC_OP_CAS = 'cas'

const	METRIC_OP_CHANGE_TTL = 'change_ttl'

const	METRIC_OP_DECR = 'decr'

const	METRIC_OP_DELETE = 'delete'

const	METRIC_OP_GET = 'get'

const	METRIC_OP_INCR = 'incr'

const	METRIC_OP_SET = 'set'

Protected Attributes inherited from BagOStuff
callable null	$asyncHandler

int[]	$attrMap = []
	Map of (ATTR_* class constant => QOS_* class constant)

bool	$debugMode = true
	Whether to send debug log entries to the SPI logger instance.

string	$keyspace
	Default keyspace; used by makeKey()

LoggerInterface	$logger

StatsdDataFactoryInterface	$stats

array< string, array >	$wrapperInfoByPrefix = []
	Cache key processing callbacks and info for metrics.

Private Member Functions
	addCasTokenFields (IDatabase $db, array $fields)
	Either append a 'castoken' field or append the fields needed to compute the CAS token.

	buildExistenceConditions (IDatabase $db, $keys, string $time)
	WHERE conditions that check for existence and liveness of keys.

	buildIncrUpsertSet (IDatabase $db, int $step, int $init, int $expiry, string $mt, int $mtUnixTs)
	SET array for handling key overwrites when a live or stale key exists.

	buildUpsertRow (IDatabase $db, $key, $serialValue, int $expiry, string $mt)
	INSERT array for handling key writes/overwrites when no live nor stale key exists.

	buildUpsertSetForOverwrite (IDatabase $db, $serialValue, int $expiry, string $mt)
	SET array for handling key overwrites when a live or stale key exists.

	dbDecodeSerialValue (IDatabase $db, $blob)

	dbEncodeSerialValue (IDatabase $db, $serialValue)

	decodeDbExpiry (IDatabase $db, string $dbExpiry)

	deleteServerObjectsExpiringBefore (IDatabase $db, $timestamp, $limit, &$keysDeletedCount=0, array $progress=null)

	doIncr ( $key, $value=1, $flags=0)

	encodeDbExpiry (IDatabase $db, int $expiry)

	fetchBlobs (array $keys, bool $getCasToken=false)

	getCasTokenFromRow (IDatabase $db, stdClass $row)
	Get a CAS token from a SELECT result row.

	getConnection ( $shardIndex)
	Get a connection to the specified database.

	getConnectionFromServerInfo ( $shardIndex, array $server)

	getConnectionViaLoadBalancer ( $shardIndex)

	getKeyLocation ( $key)
	Get the server index and table name for a given key.

	getShardServerIndexes ()

	getShardServerIndexForTag (string $tag)

	getTableNameByShard ( $index)
	Get the table name for a given shard index.

	handleDBError (DBError $exception, $shardIndex)
	Handle a DBError which occurred during a read operation.

	initSqliteDatabase (IMaintainableDatabase $db)

	isMultiPrimaryModeEnabled ()

	makeNewKeyExpiry ( $exptime, int $nowTsUnix)

	makeTimestampedModificationToken ( $mtime, IDatabase $db)
	Make a `modtoken` column value with the original time and source database server of a write.

	markServerDown (DBError $exception, $shardIndex)
	Mark a server down due to a DBConnectionError exception.

	modifyBlobs (callable $tableWriteCallback, float $mtime, array $argsByKey, int $flags, &$resByKey=[])

	modifyTableSpecificBlobsForAdd (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
	Insert key/value pairs belonging to a partition table on the the given server.

	modifyTableSpecificBlobsForCas (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
	Insert key/value pairs belonging to a partition table on the the given server.

	modifyTableSpecificBlobsForChangeTTL (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
	Update the TTL for keys belonging to a partition table on the the given server.

	modifyTableSpecificBlobsForDelete (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
	Purge/tombstone key/value pairs belonging to a partition table on the the given server.

	modifyTableSpecificBlobsForIncrInit (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
	Either increment a counter key, if it exists, or initialize it, otherwise.

	modifyTableSpecificBlobsForSet (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
	Set key/value pairs belonging to a partition table on the the given server.

	newLockingWriteSectionModificationTimestamp ( $key, &$scope)
	Get a scoped lock and modification timestamp for a critical section of reads/writes.

	occasionallyGarbageCollect (IDatabase $db)

	setAndLogDBError (DBError $exception)

	silenceTransactionProfiler ()
	Silence the transaction profiler until the return value falls out of scope.

	waitForReplication ( $shardIndex)
	Wait for replica DBs to catch up to the primary DB.

Private Attributes
const	BLOB_CASTOKEN = 2

const	BLOB_EXPIRY = 1

const	BLOB_VALUE = 0

const	GC_DELAY_SEC = 1
	How many seconds must pass before triggering a garbage collection.

const	INF_TIMESTAMP_PLACEHOLDER = '99991231235959'
	Placeholder timestamp to use for TTL_INDEFINITE that can be stored in all RDBMs types.

const	SAFE_CLOCK_BOUND_SEC = 15
	A number of seconds well above any expected clock skew.

const	SAFE_PURGE_DELAY_SEC = 3600
	A number of seconds well above any expected clock skew and replication lag.

const	SHARD_GLOBAL = 'global'

const	SHARD_LOCAL = 'local'

const	TOMB_EXPTIME = -self::SAFE_CLOCK_BOUND_SEC
	Relative seconds-to-live to use for tombstones.

const	TOMB_SERIAL = ''
	Distinct string for tombstones stored in the "serialized" value column.

Additional Inherited Members
Public Attributes inherited from BagOStuff
const	READ_LATEST = 1
	Bitfield constants for get()/getMulti(); these are only advisory.

const	READ_VERIFIED = 2

const	WRITE_ALLOW_SEGMENTS = 16

const	WRITE_BACKGROUND = 64

const	WRITE_CACHE_ONLY = 8

const	WRITE_PRUNE_SEGMENTS = 32

const	WRITE_SYNC = 4
	Bitfield constants for set()/merge(); these are only advisory.

Public Attributes inherited from Wikimedia\LightweightObjectStore\StorageAwareness
const	ATTR_DURABILITY = 2
	Durability of writes; see QOS_DURABILITY_* (higher means stronger)

const	ATTR_EMULATION = 1
	Emulation/fallback mode; see QOS_EMULATION_*; higher is better.

const	ERR_NO_RESPONSE = 1
	Storage medium failed to yield a response.

const	ERR_NONE = 0
	No storage medium error.

const	ERR_UNEXPECTED = 3
	Storage medium operation failed due to usage limitations or an I/O error.

const	ERR_UNREACHABLE = 2
	Storage medium could not be reached.

const	QOS_DURABILITY_DISK = 4
	Data is saved to disk and writes do not usually block on fsync()

const	QOS_DURABILITY_NONE = 1
	Data is never saved to begin with (blackhole store)

const	QOS_DURABILITY_RDBMS = 5
	Data is saved to disk and writes usually block on fsync(), like a standard RDBMS.

const	QOS_DURABILITY_SCRIPT = 2
	Data is lost at the end of the current web request or CLI script.

const	QOS_DURABILITY_SERVICE = 3
	Data is lost once the service storing the data restarts.

const	QOS_EMULATION_SQL = 1
	Fallback disk-based SQL store.

const	QOS_UNKNOWN = INF
	Generic "unknown" value; useful for comparisons (always "good enough")

Detailed Description

RDBMS-based caching module.

The following database sharding schemes are supported:

None; all keys map to the same shard
Hash; keys map to shards via consistent hashing
Keyspace; global keys map to the global shard and non-global keys map to the local shard

The following database replication topologies are supported:

A primary database server for each shard, all within one datacenter
A co-primary database server for each shard within each datacenter

Definition at line 52 of file SqlBagOStuff.php.

Constructor & Destructor Documentation

◆ __construct()

SqlBagOStuff::__construct ( $params )

Create a new backend instance from configuration.

The database servers must be provided by either the "server" parameter, the "servers" parameter, the "globalKeyLB" parameter, or both the "globalKeyLB"/"localKeyLB" paramters.

Parameters include:

server: Server config map for Database::factory() that describes the database to use for all key operations in the current region. This is overriden by "servers".
servers: Map of tag strings to server config maps, each for Database::factory(), describing the set of database servers on which to distribute key operations in the current region. Data is distributed among the servers via key hashing based on the server tags. Therefore, each tag represents a shard of the dataset. Tags are useful for failover using cold-standby servers and for managing shards with replica servers in multiple regions (each having different hostnames).
localKeyLB: ObjectFactory::getObjectFromSpec array yielding ILoadBalancer. This load balancer is used for local keys, e.g. those using makeKey(). This is overriden by "server" and "servers".
globalKeyLB: ObjectFactory::getObjectFromSpec array yielding ILoadBalancer. This load balancer is used for local keys, e.g. those using makeGlobalKey(). This is overriden by "server" and "servers".
multiPrimaryMode: Whether the portion of the dataset belonging to each tag/shard is replicated among one or more regions, with one "co-primary" server in each region. Queries are issued in a manner that provides Last-Write-Wins eventual consistency. This option requires the "server" or "servers" options. Only MySQL, with statment based replication (log_bin='ON' and binlog_format='STATEMENT') is supported. Also, the modtoken column must exist on the objectcache table(s). [optional]
purgePeriod: The average number of object cache writes in between garbage collection operations, where expired entries are removed from the database. Or in other words, the reciprocal of the probability of purging on any given write. If this is set to zero, purging will never be done. [optional]
purgeLimit: Maximum number of rows to purge at once. [optional]
tableName: The table name to use, default is "objectcache". [optional]
shards: The number of tables to use for data storage on each server. If greater than 1, table names are formed in the style objectcacheNNN where NNN is the shard index, between 0 and shards-1. The number of digits used in the suffix is the minimum number required to hold the largest shard index. Data is distributed among the tables via key hashing. This helps mitigate MySQL bugs 61735 and 61736. [optional]
replicaOnly: Whether to only use replica servers and only support read operations. This option requires the use of LoadBalancer ("localKeyLB"/"globalKeyLB") and should only be used by ReplicatedBagOStuff. [optional]
syncTimeout: Max seconds to wait for replica DBs to catch up for WRITE_SYNC. [optional]
writeBatchSize: Default maximum number of rows to change in each query for write operations that can be chunked into a set of smaller writes. [optional]

Parameters

array $params

Definition at line 159 of file SqlBagOStuff.php.

References $globalKeyLb, $tableName, Wikimedia\LightweightObjectStore\StorageAwareness\ATTR_DURABILITY, Wikimedia\LightweightObjectStore\StorageAwareness\ATTR_EMULATION, Wikimedia\LightweightObjectStore\StorageAwareness\QOS_DURABILITY_RDBMS, and Wikimedia\LightweightObjectStore\StorageAwareness\QOS_EMULATION_SQL.

Member Function Documentation

◆ addCasTokenFields()

SqlBagOStuff::addCasTokenFields	(	IDatabase	$db,
		array	$fields
	)

private

Either append a 'castoken' field or append the fields needed to compute the CAS token.

Parameters

IDatabase	$db
string[]	$fields	SELECT field array

Returns: string[] SELECT field array

Definition at line 1303 of file SqlBagOStuff.php.

References $type, Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\buildConcat(), and Wikimedia\Rdbms\IDatabase\getType().

Referenced by fetchBlobs(), and modifyTableSpecificBlobsForCas().

◆ buildExistenceConditions()

SqlBagOStuff::buildExistenceConditions	(	IDatabase	$db,
			$keys,
		string	$time
	)

private

WHERE conditions that check for existence and liveness of keys.

Parameters

IDatabase	$db
string[] \| string	$keys
string	$time	UNIX modification timestamp with 6 decimal places

Returns: array

Definition at line 1124 of file SqlBagOStuff.php.

References $keys, Wikimedia\Rdbms\IDatabase\addQuotes(), and Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by fetchBlobs(), modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), and modifyTableSpecificBlobsForChangeTTL().

◆ buildIncrUpsertSet()

SqlBagOStuff::buildIncrUpsertSet	(	IDatabase	$db,
		int	$step,
		int	$init,
		int	$expiry,
		string	$mt,
		int	$mtUnixTs
	)

private

SET array for handling key overwrites when a live or stale key exists.

Parameters

IDatabase	$db
int	$step	Positive counter incrementation value
int	$init	Positive initial counter value
int	$expiry	Expiration timestamp or TTL_INDEFINITE
string	$mt	Modification token
int	$mtUnixTs	UNIX timestamp of modification token

Returns: array

Definition at line 1218 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\buildIntegerCast(), Wikimedia\Rdbms\IDatabase\conditional(), dbEncodeSerialValue(), encodeDbExpiry(), isMultiPrimaryModeEnabled(), and Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by modifyTableSpecificBlobsForIncrInit().

◆ buildUpsertRow()

SqlBagOStuff::buildUpsertRow	(	IDatabase	$db,
			$key,
			$serialValue,
		int	$expiry,
		string	$mt
	)

private

INSERT array for handling key writes/overwrites when no live nor stale key exists.

Parameters

IDatabase	$db
string	$key
string \| int	$serialValue	New value
int	$expiry	Expiration timestamp or TTL_INDEFINITE
string	$mt	Modification token

Returns: array

Definition at line 1142 of file SqlBagOStuff.php.

References dbEncodeSerialValue(), encodeDbExpiry(), and isMultiPrimaryModeEnabled().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), modifyTableSpecificBlobsForIncrInit(), and modifyTableSpecificBlobsForSet().

◆ buildUpsertSetForOverwrite()

SqlBagOStuff::buildUpsertSetForOverwrite	(	IDatabase	$db,
			$serialValue,
		int	$expiry,
		string	$mt
	)

private

SET array for handling key overwrites when a live or stale key exists.

Parameters

IDatabase	$db
string \| int	$serialValue	New value
int	$expiry	Expiration timestamp or TTL_INDEFINITE
string	$mt	Modification token

Returns: array

Definition at line 1170 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\conditional(), dbEncodeSerialValue(), encodeDbExpiry(), and isMultiPrimaryModeEnabled().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), and modifyTableSpecificBlobsForSet().

◆ createTables()

SqlBagOStuff::createTables ( )

Create the shard tables on all databases (e.g.

via eval.php/shell.php)

Definition at line 1788 of file SqlBagOStuff.php.

References $numTableShards, getConnection(), getShardServerIndexes(), and getTableNameByShard().

◆ dbDecodeSerialValue()

SqlBagOStuff::dbDecodeSerialValue	(	IDatabase	$db,
			$blob
	)

private

Parameters

IDatabase	$db
Blob \| string \| int	$blob

Returns: string|int

Definition at line 1292 of file SqlBagOStuff.php.

References $blob, Wikimedia\Rdbms\IDatabase\decodeBlob(), and MediumSpecificBagOStuff\isInteger().

Referenced by fetchBlobs(), getCasTokenFromRow(), modifyTableSpecificBlobsForChangeTTL(), and modifyTableSpecificBlobsForIncrInit().

◆ dbEncodeSerialValue()

SqlBagOStuff::dbEncodeSerialValue	(	IDatabase	$db,
			$serialValue
	)

private

Parameters

IDatabase	$db
string \| int	$serialValue

Returns: string|int

Definition at line 1283 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\encodeBlob().

Referenced by buildIncrUpsertSet(), buildUpsertRow(), and buildUpsertSetForOverwrite().

◆ decodeDbExpiry()

SqlBagOStuff::decodeDbExpiry	(	IDatabase	$db,
		string	$dbExpiry
	)

private

Parameters

IDatabase	$db
string	$dbExpiry	DB timestamp of expiration

Returns: int UNIX timestamp of expiration or TTL_INDEFINITE

Definition at line 1272 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by fetchBlobs().

◆ decr()

SqlBagOStuff::decr	(	$key,
		$value = `1`,
		$flags = `0`
	)

Decrease stored value of $key by $value while preserving its TTL.

Parameters

string	$key
int	$value	Value to subtract from $key (default: 1) [optional]
int	$flags	Bit field of class WRITE_* constants [optional]

Returns: int|bool New value or false on failure

Reimplemented from BagOStuff.

Definition at line 321 of file SqlBagOStuff.php.

References doIncr().

◆ deleteAll()

SqlBagOStuff::deleteAll ( )

Delete content of shard tables in every server.

Return true if the operation is successful, false otherwise.

Returns: bool

Definition at line 1529 of file SqlBagOStuff.php.

References $numTableShards, getConnection(), getShardServerIndexes(), getTableNameByShard(), handleDBError(), and silenceTransactionProfiler().

◆ deleteObjectsExpiringBefore()

SqlBagOStuff::deleteObjectsExpiringBefore	(		$timestamp,
		callable	$progress = `null`,
			$limit = `INF`,
		string	$tag = `null`
	)

Delete all objects expiring before a certain date.

Parameters

string \| int	$timestamp	The reference date in MW or TS_UNIX format
callable \| null	$progress	Optional, a function which will be called regularly during long-running operations with the percentage progress as the first parameter. [optional]
int	$limit	Maximum number of keys to delete [default: INF]
string \| null	$tag	Tag to purge a single shard only. This is only supported when server tags are used in configuration.

Returns: bool Success; false if unimplemented

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1382 of file SqlBagOStuff.php.

References deleteServerObjectsExpiringBefore(), getConnection(), getShardServerIndexes(), getShardServerIndexForTag(), handleDBError(), and silenceTransactionProfiler().

Referenced by expireAll().

◆ deleteServerObjectsExpiringBefore()

SqlBagOStuff::deleteServerObjectsExpiringBefore	(	IDatabase	$db,
			$timestamp,
			$limit,
		&	$keysDeletedCount = `0`,
		array	$progress = `null`
	)

private

Parameters

IDatabase	$db
string \| int	$timestamp
int	$limit	Maximum number of rows to delete in total
int	&$keysDeletedCount
	null\|array{fn:callback,serversDone:int,serversTotal:int}	$progress

Exceptions

DBError

Definition at line 1430 of file SqlBagOStuff.php.

References $keys, $numTableShards, $res, Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\affectedRows(), Wikimedia\Rdbms\IDatabase\delete(), BagOStuff\getCurrentTime(), getTableNameByShard(), isMultiPrimaryModeEnabled(), SAFE_PURGE_DELAY_SEC, Wikimedia\Rdbms\IDatabase\select(), and Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by deleteObjectsExpiringBefore(), and occasionallyGarbageCollect().

◆ doAdd()

SqlBagOStuff::doAdd	(	$key,
		$value,
		$exptime = `0`,
		$flags = `0`
	)

protected

Insert an item if it does not already exist.

Parameters

string	$key
mixed	$value
int	$exptime
int	$flags	Bitfield of BagOStuff::WRITE_* constants (since 1.33)

Returns: bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 259 of file SqlBagOStuff.php.

References modifyBlobs(), and newLockingWriteSectionModificationTimestamp().

◆ doCas()

SqlBagOStuff::doCas	(	$casToken,
		$key,
		$value,
		$exptime = `0`,
		$flags = `0`
	)

protected

Check and set an item.

Parameters

mixed	$casToken
string	$key
mixed	$value
int	$exptime	Either an interval in seconds or a unix timestamp for expiry
int	$flags	Bitfield of BagOStuff::WRITE_* constants

Returns: bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 274 of file SqlBagOStuff.php.

References modifyBlobs(), and newLockingWriteSectionModificationTimestamp().

◆ doChangeTTL()

SqlBagOStuff::doChangeTTL	(	$key,
		$exptime,
		$flags
	)

protected

Parameters

string	$key
int	$exptime
int	$flags

Returns: bool

Reimplemented from MediumSpecificBagOStuff.

Definition at line 289 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doChangeTTLMulti()

SqlBagOStuff::doChangeTTLMulti	(	array	$keys,
			$exptime,
			$flags = `0`
	)

Parameters

string[]	$keys	List of keys
int	$exptime	TTL or UNIX timestamp
int	$flags	Bitfield of BagOStuff::WRITE_* constants

Returns: bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 408 of file SqlBagOStuff.php.

References $keys, BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doDelete()

SqlBagOStuff::doDelete	(	$key,
		$flags = `0`
	)

protected

Delete an item.

Parameters

string	$key
int	$flags	Bitfield of BagOStuff::WRITE_* constants

Returns: bool True if the item was deleted or not found, false on failure

Reimplemented from MediumSpecificBagOStuff.

Definition at line 248 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doDeleteMulti()

SqlBagOStuff::doDeleteMulti	(	array	$keys,
			$flags = `0`
	)

protected

Parameters

string[]	$keys	List of keys
int	$flags	Bitfield of BagOStuff::WRITE_* constants

Returns: bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 397 of file SqlBagOStuff.php.

References $keys, BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doGet()

SqlBagOStuff::doGet	(		$key,
			$flags = `0`,
		&	$casToken = `null`
	)

protected

Parameters

string	$key
int	$flags	Bitfield of BagOStuff::READ_* constants [optional]
mixed \| null	&$casToken	cas() token if MediumSpecificBagOStuff::PASS_BY_REF [returned]

Returns: mixed Returns false on failure or if the item does not exist

Reimplemented from MediumSpecificBagOStuff.

Definition at line 216 of file SqlBagOStuff.php.

References BLOB_CASTOKEN, fetchBlobs(), unserialize(), and MediumSpecificBagOStuff\updateOpStats().

◆ doGetMulti()

SqlBagOStuff::doGetMulti	(	array	$keys,
			$flags = `0`
	)

protected

Get an associative array containing the item for each of the keys that have items.

Parameters

string[]	$keys	List of keys
int	$flags	Bitfield; supports READ_LATEST [optional]

Returns: array Map of (key => value) for existing keys; preserves the order of $keys

Reimplemented from MediumSpecificBagOStuff.

Definition at line 356 of file SqlBagOStuff.php.

References $keys, BLOB_VALUE, fetchBlobs(), unserialize(), and MediumSpecificBagOStuff\updateOpStats().

◆ doIncr()

SqlBagOStuff::doIncr	(	$key,
		$value = `1`,
		$flags = `0`
	)

private

Definition at line 325 of file SqlBagOStuff.php.

References BLOB_VALUE, fetchBlobs(), MediumSpecificBagOStuff\isInteger(), modifyBlobs(), and newLockingWriteSectionModificationTimestamp().

Referenced by decr(), and incr().

◆ doLock()

SqlBagOStuff::doLock	(	$key,
		$timeout = `6`,
		$exptime = `6`
	)

See also: MediumSpecificBagOStuff::lock()

Parameters

string	$key
int	$timeout	Lock wait timeout; 0 for non-blocking [optional]
int	$exptime	Lock time-to-live 1 day maximum [optional]

Returns: float|null UNIX timestamp of acquisition; null on failure

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1547 of file SqlBagOStuff.php.

References getConnection(), getKeyLocation(), handleDBError(), and silenceTransactionProfiler().

◆ doSet()

SqlBagOStuff::doSet	(	$key,
		$value,
		$exptime = `0`,
		$flags = `0`
	)

protected

Set an item.

Parameters

string	$key
mixed	$value
int	$exptime	Either an interval in seconds or a unix timestamp for expiry
int	$flags	Bitfield of BagOStuff::WRITE_* constants

Returns: bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 237 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doSetMulti()

SqlBagOStuff::doSetMulti	(	array	$data,
			$exptime = `0`,
			$flags = `0`
	)

protected

Parameters

mixed[]	$data	Map of (key => value)
int	$exptime	Either an interval in seconds or a unix timestamp for expiry
int	$flags	Bitfield of BagOStuff::WRITE_* constants

Returns: bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 381 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doUnlock()

SqlBagOStuff::doUnlock ( $key )

See also: MediumSpecificBagOStuff::unlock()

Parameters

string $key

Returns: bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1568 of file SqlBagOStuff.php.

References getConnection(), getKeyLocation(), handleDBError(), and silenceTransactionProfiler().

◆ encodeDbExpiry()

SqlBagOStuff::encodeDbExpiry	(	IDatabase	$db,
		int	$expiry
	)

private

Parameters

IDatabase	$db
int	$expiry	UNIX timestamp of expiration or TTL_INDEFINITE

Returns: string

Definition at line 1259 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by buildIncrUpsertSet(), buildUpsertRow(), buildUpsertSetForOverwrite(), and modifyTableSpecificBlobsForChangeTTL().

◆ expireAll()

SqlBagOStuff::expireAll ( )

Definition at line 1378 of file SqlBagOStuff.php.

References deleteObjectsExpiringBefore(), and BagOStuff\getCurrentTime().

◆ fetchBlobs()

SqlBagOStuff::fetchBlobs	(	array	$keys,
		bool	$getCasToken = `false`
	)

private

Parameters

string[]	$keys
bool	$getCasToken	Whether to get a CAS token

Returns: array<string,array|null> Order-preserved map of (key => (value,expiry,token) or null)

Definition at line 507 of file SqlBagOStuff.php.

References $keys, $res, addCasTokenFields(), buildExistenceConditions(), dbDecodeSerialValue(), MediumSpecificBagOStuff\debug(), decodeDbExpiry(), getCasTokenFromRow(), getConnection(), BagOStuff\getCurrentTime(), getKeyLocation(), handleDBError(), and silenceTransactionProfiler().

Referenced by doGet(), doGetMulti(), and doIncr().

◆ getCasTokenFromRow()

SqlBagOStuff::getCasTokenFromRow	(	IDatabase	$db,
		stdClass	$row
	)

private

Get a CAS token from a SELECT result row.

Parameters

IDatabase	$db
stdClass	$row	A row for a key

Returns: string CAS token

Definition at line 1337 of file SqlBagOStuff.php.

References dbDecodeSerialValue().

Referenced by fetchBlobs(), and modifyTableSpecificBlobsForCas().

◆ getConnection()

SqlBagOStuff::getConnection ( $shardIndex )

private

Get a connection to the specified database.

Parameters

int | string $shardIndex Server index or self::SHARD_LOCAL/selfSHARD_GLOBAL

Returns: IMaintainableDatabase

Exceptions

DBConnectionError
UnexpectedValueException

Definition at line 427 of file SqlBagOStuff.php.

References getConnectionFromServerInfo(), getConnectionViaLoadBalancer(), and BagOStuff\getCurrentTime().

Referenced by createTables(), deleteAll(), deleteObjectsExpiringBefore(), doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ getConnectionFromServerInfo()

SqlBagOStuff::getConnectionFromServerInfo	(		$shardIndex,
		array	$server
	)

private

Parameters

int	$shardIndex
array	$server	Server config map

Returns: IMaintainableDatabase

Exceptions

DBError

Definition at line 1676 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\getType(), initSqliteDatabase(), and Wikimedia\Rdbms\IDatabase\tableExists().

Referenced by getConnection().

◆ getConnectionViaLoadBalancer()

SqlBagOStuff::getConnectionViaLoadBalancer ( $shardIndex )

private

Parameters

string $shardIndex self::SHARD_LOCAL/selfSHARD_GLOBAL

Returns: IMaintainableDatabase

Exceptions

DBError

Definition at line 1652 of file SqlBagOStuff.php.

References $globalKeyLb, DB_PRIMARY, DB_REPLICA, Wikimedia\Rdbms\ILoadBalancer\getMaintenanceConnectionRef(), and SHARD_LOCAL.

Referenced by getConnection().

◆ getKeyLocation()

SqlBagOStuff::getKeyLocation ( $key )

private

Get the server index and table name for a given key.

Parameters

string $key

Returns: array (server index or self::SHARD_LOCAL/selfSHARD_GLOBAL, table name)

Definition at line 459 of file SqlBagOStuff.php.

References $globalKeyLb, $numTableShards, $serverTags, getTableNameByShard(), and SHARD_LOCAL.

Referenced by doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ getShardServerIndexes()

SqlBagOStuff::getShardServerIndexes ( )

private

Returns: string[]|int[] List of server indexes or self::SHARD_LOCAL/selfSHARD_GLOBAL

Definition at line 1804 of file SqlBagOStuff.php.

References SHARD_GLOBAL, and SHARD_LOCAL.

Referenced by createTables(), deleteAll(), and deleteObjectsExpiringBefore().

◆ getShardServerIndexForTag()

SqlBagOStuff::getShardServerIndexForTag ( string $tag )

private

Parameters

string $tag

Returns: int Server index for use with getConnection()

Exceptions

InvalidArgumentException If tag is unknown

Definition at line 1827 of file SqlBagOStuff.php.

Referenced by deleteObjectsExpiringBefore().

◆ getTableNameByShard()

SqlBagOStuff::getTableNameByShard ( $index )

private

Get the table name for a given shard index.

Parameters

int | null $index

Returns: string

Definition at line 492 of file SqlBagOStuff.php.

References $tableName.

Referenced by createTables(), deleteAll(), deleteServerObjectsExpiringBefore(), and getKeyLocation().

◆ handleDBError()

SqlBagOStuff::handleDBError	(	DBError	$exception,
			$shardIndex
	)

private

Handle a DBError which occurred during a read operation.

Parameters

DBError	$exception
int \| string	$shardIndex	Server index or self::SHARD_LOCAL/selfSHARD_GLOBAL

Definition at line 1709 of file SqlBagOStuff.php.

References markServerDown(), and setAndLogDBError().

Referenced by deleteAll(), deleteObjectsExpiringBefore(), doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ incr()

SqlBagOStuff::incr	(	$key,
		$value = `1`,
		$flags = `0`
	)

Increase stored value of $key by $value while preserving its TTL.

Parameters

string	$key	Key to increase
int	$value	Value to add to $key (default: 1) [optional]
int	$flags	Bit field of class WRITE_* constants [optional]

Returns: int|bool New value or false on failure

Reimplemented from BagOStuff.

Definition at line 317 of file SqlBagOStuff.php.

References doIncr().

◆ incrWithInit()

SqlBagOStuff::incrWithInit	(	$key,
		$exptime,
		$value = `1`,
		$init = `null`,
		$flags = `0`
	)

Increase the value of the given key (no TTL change) if it exists or create it otherwise.

This will create the key with the value $init and TTL $exptime instead if not present. Callers should make sure that both ($init - $value) and $exptime are invariants for all operations to any given key. The value of $init should be at least that of $value.

Parameters

string	$key	Key built via makeKey() or makeGlobalKey()
int	$exptime	Time-to-live (in seconds) or a UNIX timestamp expiration
int	$value	Amount to increase the key value by [default: 1]
int \| null	$init	Value to initialize the key to if it does not exist [default: $value]
int	$flags	Bit field of class WRITE_* constants [optional]

Returns: int|bool New value or false on failure

Since: 1.24

Reimplemented from MediumSpecificBagOStuff.

Definition at line 300 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ initSqliteDatabase()

SqlBagOStuff::initSqliteDatabase ( IMaintainableDatabase $db )

private

Parameters

IMaintainableDatabase $db

Exceptions

DBError

Definition at line 1759 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\addIdentifierQuotes(), Wikimedia\Rdbms\IDatabase\endAtomic(), Wikimedia\Rdbms\IDatabase\query(), Wikimedia\Rdbms\IDatabase\rollback(), Wikimedia\Rdbms\IDatabase\startAtomic(), Wikimedia\Rdbms\IDatabase\tableExists(), Wikimedia\Rdbms\IMaintainableDatabase\tableName(), and Wikimedia\Rdbms\IDatabase\tablePrefix().

Referenced by getConnectionFromServerInfo().

◆ isMultiPrimaryModeEnabled()

SqlBagOStuff::isMultiPrimaryModeEnabled ( )

private

Returns: bool Whether queries should be multi-primary safe

Definition at line 1842 of file SqlBagOStuff.php.

Referenced by buildIncrUpsertSet(), buildUpsertRow(), buildUpsertSetForOverwrite(), deleteServerObjectsExpiringBefore(), modifyTableSpecificBlobsForChangeTTL(), and modifyTableSpecificBlobsForDelete().

◆ makeKeyInternal()

SqlBagOStuff::makeKeyInternal	(	$keyspace,
		$components
	)

Make a cache key for the given keyspace and components.

Long components might be converted to respective hashes due to size constraints. In extreme cases, all of them might be combined into a single hash component.

Access: internal: This method should not be used outside of BagOStuff (since 1.36)

Parameters

string	$keyspace	Keyspace component
string[] \| int[]	$components	Key components (key collection name first)

Returns: string Keyspace-prepended list of encoded components as a colon-separated value

Since: 1.27

Reimplemented from BagOStuff.

Definition at line 1585 of file SqlBagOStuff.php.

References BagOStuff\$keyspace.

◆ makeNewKeyExpiry()

SqlBagOStuff::makeNewKeyExpiry	(		$exptime,
		int	$nowTsUnix
	)

private

Parameters

int	$exptime	Relative or absolute expiration
int	$nowTsUnix	Current UNIX timestamp

Returns: int UNIX timestamp or TTL_INDEFINITE

Definition at line 1032 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\getExpirationAsTimestamp().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), modifyTableSpecificBlobsForIncrInit(), and modifyTableSpecificBlobsForSet().

◆ makeTimestampedModificationToken()

SqlBagOStuff::makeTimestampedModificationToken	(		$mtime,
		IDatabase	$db
	)

private

Make a modtoken column value with the original time and source database server of a write.

Parameters

int \| float	$mtime	UNIX modification timestamp
IDatabase	$db	Handle to the primary database server sourcing the write

Returns: string String of the form "<SECONDS_SOURCE><MICROSECONDS>", where SECONDS_SOURCE is "<35 bit seconds portion of UNIX time><32 bit database server ID>" as 13 base 36 chars, and MICROSECONDS is "<20 bit microseconds portion of UNIX time>" as 4 base 36 chars

Definition at line 1083 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\getServerName(), and Wikimedia\Rdbms\IDatabase\getTopologyBasedServerId().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), modifyTableSpecificBlobsForIncrInit(), and modifyTableSpecificBlobsForSet().

◆ markServerDown()

SqlBagOStuff::markServerDown	(	DBError	$exception,
			$shardIndex
	)

private

Mark a server down due to a DBConnectionError exception.

Parameters

DBError	$exception
int \| string	$shardIndex	Server index or self::SHARD_LOCAL/selfSHARD_GLOBAL

Definition at line 1737 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime().

Referenced by handleDBError().

◆ modifyBlobs()

SqlBagOStuff::modifyBlobs	(	callable	$tableWriteCallback,
		float	$mtime,
		array	$argsByKey,
		int	$flags,
		&	$resByKey = `[]`
	)

private

Parameters

callable	$tableWriteCallback	Callback the takes the following arguments: IDatabase instance Partition table name string UNIX modification timestamp Map of (key => list of arguments) for keys belonging to the server/table partition Map of (key => result) [returned]
float	$mtime	UNIX modification timestamp
	array<string,array>	$argsByKey Map of (key => list of arguments)
int	$flags	Bitfield of BagOStuff::WRITE_* constants
	array<string,mixed>	&$resByKey Order-preserved map of (key => result) [returned]

Returns: bool Whether all keys were processed

Parameters

-taint $argsByKey none

Definition at line 582 of file SqlBagOStuff.php.

References $args, $success, BagOStuff\fieldHasFlags(), getConnection(), getKeyLocation(), handleDBError(), occasionallyGarbageCollect(), silenceTransactionProfiler(), and waitForReplication().

Referenced by doAdd(), doCas(), doChangeTTL(), doChangeTTLMulti(), doDelete(), doDeleteMulti(), doIncr(), doSet(), doSetMulti(), and incrWithInit().

◆ modifyTableSpecificBlobsForAdd()

SqlBagOStuff::modifyTableSpecificBlobsForAdd	(	IDatabase	$db,
		string	$ptable,
		float	$mtime,
		array	$argsByKey,
		array &	$resByKey
	)

private

Insert key/value pairs belonging to a partition table on the the given server.

If the current row for a key exists and has an integral UNIX timestamp of expiration greater than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Aquisition of advisory key locks must be handled by calling functions.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp.

Parameters

IDatabase	$db	Handle to the database server where the argument keys belong
string	$ptable	Name of the partition table where the argument keys belong
float	$mtime	UNIX modification timestamp
	array<string,array>	$argsByKey Non-empty (key => (value,exptime)) map
	array<string,mixed>	&$resByKey Map of (key => result) prefilled with false [returned]

Exceptions

DBError

Definition at line 768 of file SqlBagOStuff.php.

References buildExistenceConditions(), buildUpsertRow(), buildUpsertSetForOverwrite(), MediumSpecificBagOStuff\getSerialized(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\selectFieldValues(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForCas()

SqlBagOStuff::modifyTableSpecificBlobsForCas	(	IDatabase	$db,
		string	$ptable,
		float	$mtime,
		array	$argsByKey,
		array &	$resByKey
	)

private

Insert key/value pairs belonging to a partition table on the the given server.

If the current row for a key exists, has an integral UNIX timestamp of expiration greater than that of the provided modification timestamp, and the CAS token does not match, then the write to that key will be aborted with a "false" result. Aquisition of advisory key locks must be handled by calling functions.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp.

Parameters

IDatabase	$db	Handle to the database server where the argument keys belong
string	$ptable	Name of the partition table where the argument keys belong
float	$mtime	UNIX modification timestamp
	array<string,array>	$argsByKey Non-empty (key => (value, exptime, CAS token)) map
	array<string,mixed>	&$resByKey Map of (key => result) prefilled with false [returned]

Exceptions

DBError

Definition at line 832 of file SqlBagOStuff.php.

References $res, addCasTokenFields(), buildExistenceConditions(), buildUpsertRow(), buildUpsertSetForOverwrite(), getCasTokenFromRow(), MediumSpecificBagOStuff\getSerialized(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\select(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForChangeTTL()

SqlBagOStuff::modifyTableSpecificBlobsForChangeTTL	(	IDatabase	$db,
		string	$ptable,
		float	$mtime,
		array	$argsByKey,
		array &	$resByKey
	)

private

Update the TTL for keys belonging to a partition table on the the given server.

If no current row for a key exists or the current row has an integral UNIX timestamp of expiration less than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp.

Parameters

IDatabase	$db	Handle to the database server where the argument keys belong
string	$ptable	Name of the partition table where the argument keys belong
float	$mtime	UNIX modification timestamp
	array<string,array>	$argsByKey Non-empty (key => (exptime)) map
	array<string,mixed>	&$resByKey Map of (key => result) prefilled with false [returned]

Exceptions

DBError

Definition at line 905 of file SqlBagOStuff.php.

References $args, $res, Wikimedia\Rdbms\IDatabase\affectedRows(), buildExistenceConditions(), buildUpsertRow(), buildUpsertSetForOverwrite(), dbDecodeSerialValue(), encodeDbExpiry(), isMultiPrimaryModeEnabled(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\select(), Wikimedia\Rdbms\IDatabase\update(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForDelete()

SqlBagOStuff::modifyTableSpecificBlobsForDelete	(	IDatabase	$db,
		string	$ptable,
		float	$mtime,
		array	$argsByKey,
		array &	$resByKey
	)

private

Purge/tombstone key/value pairs belonging to a partition table on the the given server.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp, an empty value, and an expiration timestamp dated slightly before the new modification timestamp.

Parameters

IDatabase	$db	Handle to the database server where the argument keys belong
string	$ptable	Name of the partition table where the argument keys belong
float	$mtime	UNIX modification timestamp
	array<string,array>	$argsByKey Non-empty (key => []) map
	array<string,mixed>	&$resByKey Map of (key => result) prefilled with false [returned]

Exceptions

DBError

Definition at line 714 of file SqlBagOStuff.php.

References buildUpsertRow(), buildUpsertSetForOverwrite(), Wikimedia\Rdbms\IDatabase\delete(), isMultiPrimaryModeEnabled(), makeNewKeyExpiry(), makeTimestampedModificationToken(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForIncrInit()

SqlBagOStuff::modifyTableSpecificBlobsForIncrInit	(	IDatabase	$db,
		string	$ptable,
		float	$mtime,
		array	$argsByKey,
		array &	$resByKey
	)

private

Either increment a counter key, if it exists, or initialize it, otherwise.

If no current row for a key exists or the current row has an integral UNIX timestamp of expiration less than that of the provided modification timestamp, then the key row will be set to the initial value. Otherwise, the current row will be incremented.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully initialized key rows will be assigned a new modification token/timestamp.

Parameters

IDatabase	$db	Handle to the database server where the argument keys belong
string	$ptable	Name of the partition table where the argument keys belong
float	$mtime	UNIX modification timestamp
	array<string,array>	$argsByKey Non-empty (key => (step, init, exptime) map
	array<string,mixed>	&$resByKey Map of (key => result) prefilled with false [returned]

Exceptions

DBError

Definition at line 983 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\affectedRows(), buildIncrUpsertSet(), buildUpsertRow(), dbDecodeSerialValue(), Wikimedia\Rdbms\IDatabase\endAtomic(), MediumSpecificBagOStuff\isInteger(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\selectRow(), Wikimedia\Rdbms\IDatabase\startAtomic(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForSet()

SqlBagOStuff::modifyTableSpecificBlobsForSet	(	IDatabase	$db,
		string	$ptable,
		float	$mtime,
		array	$argsByKey,
		array &	$resByKey
	)

private

Set key/value pairs belonging to a partition table on the the given server.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token using the provided timestamp.

Parameters

IDatabase	$db	Handle to the database server where the argument keys belong
string	$ptable	Name of the partition table where the argument keys belong
float	$mtime	UNIX modification timestamp
	array<string,array>	$argsByKey Non-empty (key => (value,exptime)) map
	array<string,mixed>	&$resByKey Map of (key => result) for succesful writes [returned]

Exceptions

DBError

Definition at line 652 of file SqlBagOStuff.php.

References buildUpsertRow(), buildUpsertSetForOverwrite(), MediumSpecificBagOStuff\getSerialized(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\replace(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ newLockingWriteSectionModificationTimestamp()

SqlBagOStuff::newLockingWriteSectionModificationTimestamp	(		$key,
		&	$scope
	)

private

Get a scoped lock and modification timestamp for a critical section of reads/writes.

This is used instead of BagOStuff::getCurrentTime() for certain writes (such as "add", "incr", and "cas"), for which we want to support tight race conditions where the same key is repeatedly written to by multiple web servers that each get to see the previous value, act on it, and modify it in some way.

It is assumed that this method is normally only invoked from the primary datacenter. A lock is acquired on the primary server of the local datacenter in order to avoid race conditions within the critical section. The clock on the SQL server is used to get the modification timestamp in order to minimize issues with clock drift between web servers; thus key writes will not be rejected due to some web servers having lagged clocks.

Parameters

string	$key
	?ScopedCallback	&$scope Unlocker callback; null on failure [returned]

Returns: string|null UNIX timestamp with 6 decimal places; null on failure

Definition at line 1062 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\lock(), and MediumSpecificBagOStuff\unlock().

Referenced by doAdd(), doCas(), and doIncr().

◆ occasionallyGarbageCollect()

SqlBagOStuff::occasionallyGarbageCollect ( IDatabase $db )

private

Parameters

IDatabase $db

Exceptions

DBError

Definition at line 1352 of file SqlBagOStuff.php.

References BagOStuff\$asyncHandler, deleteServerObjectsExpiringBefore(), and BagOStuff\getCurrentTime().

Referenced by modifyBlobs().

◆ serialize()

SqlBagOStuff::serialize ( $value )

protected

Parameters

mixed $value

Returns: string|int|false String/integer representation

Note: Special handling is usually needed for integers so incr()/decr() work

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1611 of file SqlBagOStuff.php.

References serialize().

◆ setAndLogDBError()

SqlBagOStuff::setAndLogDBError ( DBError $exception )

private

Parameters

DBError $exception

Definition at line 1720 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\setLastError().

Referenced by handleDBError(), and waitForReplication().

◆ silenceTransactionProfiler()

SqlBagOStuff::silenceTransactionProfiler ( )

private

Silence the transaction profiler until the return value falls out of scope.

Returns: ScopedCallback|null

Definition at line 1890 of file SqlBagOStuff.php.

Referenced by deleteAll(), deleteObjectsExpiringBefore(), doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ unserialize()

SqlBagOStuff::unserialize ( $value )

protected

Parameters

string | int | false $value

Returns: mixed Original value or false on error

Note: Special handling is usually needed for integers so incr()/decr() work

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1625 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\isInteger(), and unserialize().

Referenced by doGet(), doGetMulti(), and unserialize().

◆ waitForReplication()

SqlBagOStuff::waitForReplication ( $shardIndex )

private

Wait for replica DBs to catch up to the primary DB.

Parameters

int | string $shardIndex Server index or self::SHARD_LOCAL/selfSHARD_GLOBAL

Returns: bool Success

Definition at line 1852 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\$busyCallbacks, $globalKeyLb, MediumSpecificBagOStuff\$syncTimeout, Wikimedia\Rdbms\ILoadBalancer\getPrimaryPos(), setAndLogDBError(), and SHARD_LOCAL.

Referenced by modifyBlobs().

Member Data Documentation

◆ $connFailureErrors

Exception [] SqlBagOStuff::$connFailureErrors = []

protected

Map of (shard index => Exception)

Definition at line 84 of file SqlBagOStuff.php.

◆ $connFailureTimes

float [] SqlBagOStuff::$connFailureTimes = []

protected

Map of (shard index => UNIX timestamps)

Definition at line 82 of file SqlBagOStuff.php.

◆ $conns

IMaintainableDatabase [] SqlBagOStuff::$conns

protected

Map of (shard index => DB handle)

Definition at line 80 of file SqlBagOStuff.php.

◆ $globalKeyLb

ILoadBalancer null SqlBagOStuff::$globalKeyLb

protected

Definition at line 56 of file SqlBagOStuff.php.

Referenced by __construct(), getConnectionViaLoadBalancer(), getKeyLocation(), and waitForReplication().

◆ $lastGarbageCollect

int SqlBagOStuff::$lastGarbageCollect = 0

protected

UNIX timestamp.

Definition at line 63 of file SqlBagOStuff.php.

◆ $localKeyLb

ILoadBalancer null SqlBagOStuff::$localKeyLb

protected

Definition at line 54 of file SqlBagOStuff.php.

◆ $multiPrimaryModeType

string null SqlBagOStuff::$multiPrimaryModeType

protected

Multi-primary mode DB type ("mysql",...); null if not enabled.

Definition at line 77 of file SqlBagOStuff.php.

◆ $numTableShards

int SqlBagOStuff::$numTableShards = 1

protected

Number of table shards to use on each server.

Definition at line 69 of file SqlBagOStuff.php.

Referenced by createTables(), deleteAll(), deleteServerObjectsExpiringBefore(), and getKeyLocation().

◆ $purgeLimit

int SqlBagOStuff::$purgeLimit = 100

protected

Max expired rows to purge during randomized garbage collection.

Definition at line 67 of file SqlBagOStuff.php.

◆ $purgePeriod

int SqlBagOStuff::$purgePeriod = 10

protected

Average number of writes required to trigger garbage collection.

Definition at line 65 of file SqlBagOStuff.php.

◆ $replicaOnly

bool SqlBagOStuff::$replicaOnly

protected

Whether to use replicas instead of primaries (if using LoadBalancer)

Definition at line 75 of file SqlBagOStuff.php.

◆ $serverInfos

array [] SqlBagOStuff::$serverInfos = []

protected

(server index => server config)

Definition at line 59 of file SqlBagOStuff.php.

◆ $serverTags

string [] SqlBagOStuff::$serverTags = []

protected

(server index => tag/host name)

Definition at line 61 of file SqlBagOStuff.php.

Referenced by getKeyLocation().

◆ $tableName

string SqlBagOStuff::$tableName = 'objectcache'

protected

Definition at line 73 of file SqlBagOStuff.php.

Referenced by __construct(), and getTableNameByShard().

◆ $writeBatchSize

int SqlBagOStuff::$writeBatchSize = 100

protected

Definition at line 71 of file SqlBagOStuff.php.

◆ BLOB_CASTOKEN

const SqlBagOStuff::BLOB_CASTOKEN = 2

private

Definition at line 102 of file SqlBagOStuff.php.

Referenced by doGet().

◆ BLOB_EXPIRY

const SqlBagOStuff::BLOB_EXPIRY = 1

private

Definition at line 101 of file SqlBagOStuff.php.

◆ BLOB_VALUE

const SqlBagOStuff::BLOB_VALUE = 0

private

Definition at line 100 of file SqlBagOStuff.php.

Referenced by doGetMulti(), and doIncr().

◆ GC_DELAY_SEC

const SqlBagOStuff::GC_DELAY_SEC = 1

private

How many seconds must pass before triggering a garbage collection.

Definition at line 98 of file SqlBagOStuff.php.

◆ INF_TIMESTAMP_PLACEHOLDER

const SqlBagOStuff::INF_TIMESTAMP_PLACEHOLDER = '99991231235959'

private

Placeholder timestamp to use for TTL_INDEFINITE that can be stored in all RDBMs types.

We use BINARY(14) for MySQL, BLOB for Sqlite, and TIMESTAMPZ for Postgres (which goes up to 294276 AD). The last second of the year 9999 can be stored in all these cases. https://www.postgresql.org/docs/9.0/datatype-datetime.html

Definition at line 110 of file SqlBagOStuff.php.

◆ SAFE_CLOCK_BOUND_SEC

const SqlBagOStuff::SAFE_CLOCK_BOUND_SEC = 15

private

A number of seconds well above any expected clock skew.

Definition at line 90 of file SqlBagOStuff.php.

◆ SAFE_PURGE_DELAY_SEC

const SqlBagOStuff::SAFE_PURGE_DELAY_SEC = 3600

private

A number of seconds well above any expected clock skew and replication lag.

Definition at line 92 of file SqlBagOStuff.php.

Referenced by deleteServerObjectsExpiringBefore().

◆ SHARD_GLOBAL

const SqlBagOStuff::SHARD_GLOBAL = 'global'

private

Definition at line 87 of file SqlBagOStuff.php.

Referenced by getShardServerIndexes().

◆ SHARD_LOCAL

const SqlBagOStuff::SHARD_LOCAL = 'local'

private

Definition at line 86 of file SqlBagOStuff.php.

Referenced by getConnectionViaLoadBalancer(), getKeyLocation(), getShardServerIndexes(), and waitForReplication().

◆ TOMB_EXPTIME

const SqlBagOStuff::TOMB_EXPTIME = -self::SAFE_CLOCK_BOUND_SEC

private

Relative seconds-to-live to use for tombstones.

Definition at line 96 of file SqlBagOStuff.php.

◆ TOMB_SERIAL

const SqlBagOStuff::TOMB_SERIAL = ''

private

Distinct string for tombstones stored in the "serialized" value column.

Definition at line 94 of file SqlBagOStuff.php.

The documentation for this class was generated from the following file:

includes/objectcache/SqlBagOStuff.php

Public Member Functions

Protected Member Functions

Protected Attributes

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ addCasTokenFields()

◆ buildExistenceConditions()

◆ buildIncrUpsertSet()

◆ buildUpsertRow()

◆ buildUpsertSetForOverwrite()

◆ createTables()

◆ dbDecodeSerialValue()

◆ dbEncodeSerialValue()

◆ decodeDbExpiry()

◆ decr()

◆ deleteAll()

◆ deleteObjectsExpiringBefore()

◆ deleteServerObjectsExpiringBefore()

◆ doAdd()

◆ doCas()

◆ doChangeTTL()

◆ doChangeTTLMulti()

◆ doDelete()

◆ doDeleteMulti()

◆ doGet()

◆ doGetMulti()

◆ doIncr()

◆ doLock()

◆ doSet()

◆ doSetMulti()

◆ doUnlock()

◆ encodeDbExpiry()

◆ expireAll()

◆ fetchBlobs()

◆ getCasTokenFromRow()

◆ getConnection()

◆ getConnectionFromServerInfo()

◆ getConnectionViaLoadBalancer()

◆ getKeyLocation()

◆ getShardServerIndexes()

◆ getShardServerIndexForTag()

◆ getTableNameByShard()

◆ handleDBError()

◆ incr()

◆ incrWithInit()

◆ initSqliteDatabase()

◆ isMultiPrimaryModeEnabled()

◆ makeKeyInternal()

◆ makeNewKeyExpiry()

◆ makeTimestampedModificationToken()

◆ markServerDown()

◆ modifyBlobs()

◆ modifyTableSpecificBlobsForAdd()

◆ modifyTableSpecificBlobsForCas()

◆ modifyTableSpecificBlobsForChangeTTL()

◆ modifyTableSpecificBlobsForDelete()

◆ modifyTableSpecificBlobsForIncrInit()

◆ modifyTableSpecificBlobsForSet()

◆ newLockingWriteSectionModificationTimestamp()

◆ occasionallyGarbageCollect()

◆ serialize()

◆ setAndLogDBError()

◆ silenceTransactionProfiler()

◆ unserialize()

◆ waitForReplication()

Member Data Documentation

◆ $connFailureErrors

◆ $connFailureTimes

◆ $conns

◆ $globalKeyLb

◆ $lastGarbageCollect

◆ $localKeyLb

◆ $multiPrimaryModeType

◆ $numTableShards

◆ $purgeLimit