MediaWiki  master
SqlBagOStuff Class Reference

RDBMS-based caching module. More...

Inheritance diagram for SqlBagOStuff:
Collaboration diagram for SqlBagOStuff:

Public Member Functions

 __construct ( $params)
 Create a new backend instance from configuration. More...
 
 createTables ()
 Create the shard tables on all databases (e.g. More...
 
 decr ( $key, $value=1, $flags=0)
 Decrease stored value of $key by $value while preserving its TTL. More...
 
 deleteAll ()
 Delete content of shard tables in every server. More...
 
 deleteObjectsExpiringBefore ( $timestamp, callable $progress=null, $limit=INF, string $tag=null)
 Delete all objects expiring before a certain date. More...
 
 doChangeTTLMulti (array $keys, $exptime, $flags=0)
 
 doLock ( $key, $timeout=6, $exptime=6)
 
 doUnlock ( $key)
 
 expireAll ()
 
 incr ( $key, $value=1, $flags=0)
 Increase stored value of $key by $value while preserving its TTL. More...
 
 incrWithInit ( $key, $exptime, $value=1, $init=null, $flags=0)
 Increase the value of the given key (no TTL change) if it exists or create it otherwise. More...
 
 makeKeyInternal ( $keyspace, $components)
 Make a cache key for the given keyspace and components. More...
 
- Public Member Functions inherited from MediumSpecificBagOStuff
 __construct (array $params=[])
 
 add ( $key, $value, $exptime=0, $flags=0)
 Insert an item if it does not already exist. More...
 
 addBusyCallback (callable $workCallback)
 Let a callback be run to avoid wasting time on special blocking calls. More...
 
 changeTTL ( $key, $exptime=0, $flags=0)
 Change the expiration on a key if it exists. More...
 
 changeTTLMulti (array $keys, $exptime, $flags=0)
 Change the expiration of multiple keys that exist. More...
 
 clearLastError ()
 Clear the "last error" registry. More...
 
 delete ( $key, $flags=0)
 Delete an item. More...
 
 deleteMulti (array $keys, $flags=0)
 Batch deletion. More...
 
 get ( $key, $flags=0)
 Get an item with the given key. More...
 
 getLastError ()
 Get the "last error" registered; clearLastError() should be called manually. More...
 
 getMulti (array $keys, $flags=0)
 Get an associative array containing the item for each of the keys that have items. More...
 
 getQoS ( $flag)
 
 getSegmentationSize ()
 
 getSegmentedValueMaxSize ()
 
 lock ( $key, $timeout=6, $exptime=6, $rclass='')
 
 makeGlobalKey ( $collection,... $components)
 Make a cache key for the default keyspace and given components. More...
 
 makeKey ( $collection,... $components)
 Make a cache key for the global keyspace and given components. More...
 
 merge ( $key, callable $callback, $exptime=0, $attempts=10, $flags=0)
 Merge changes into the existing cache value (possibly creating a new one) More...
 
 set ( $key, $value, $exptime=0, $flags=0)
 Set an item. More...
 
 setMulti (array $valueByKey, $exptime=0, $flags=0)
 Batch insertion/replace. More...
 
 setNewPreparedValues (array $valueByKey)
 Make a "generic" reversible cache key from the given components. More...
 
 unlock ( $key)
 Release an advisory lock on a key string. More...
 
- Public Member Functions inherited from BagOStuff
 getCurrentTime ()
 
 getLogger ()
 
 getScopedLock ( $key, $timeout=6, $expiry=30, $rclass='')
 Get a lightweight exclusive self-unlocking lock. More...
 
 getWithSetCallback ( $key, $exptime, $callback, $flags=0)
 Get an item with the given key, regenerating and setting it if not found. More...
 
 isKeyGlobal ( $key)
 Check whether a cache key is in the global keyspace. More...
 
 registerWrapperInfoForStats (string $prefixComponent, string $statsGroup, callable $collectionCallback)
 Register info about a caching layer class that uses BagOStuff as a backing store. More...
 
 setLogger (LoggerInterface $logger)
 
 setMockTime (&$time)
 

Protected Member Functions

 doAdd ( $key, $value, $exptime=0, $flags=0)
 Insert an item if it does not already exist. More...
 
 doCas ( $casToken, $key, $value, $exptime=0, $flags=0)
 Check and set an item. More...
 
 doChangeTTL ( $key, $exptime, $flags)
 
 doDelete ( $key, $flags=0)
 Delete an item. More...
 
 doDeleteMulti (array $keys, $flags=0)
 
 doGet ( $key, $flags=0, &$casToken=null)
 
 doGetMulti (array $keys, $flags=0)
 Get an associative array containing the item for each of the keys that have items. More...
 
 doSet ( $key, $value, $exptime=0, $flags=0)
 Set an item. More...
 
 doSetMulti (array $data, $exptime=0, $flags=0)
 
 serialize ( $value)
 
 unserialize ( $value)
 
- Protected Member Functions inherited from MediumSpecificBagOStuff
 cas ( $casToken, $key, $value, $exptime=0, $flags=0)
 Check and set an item. More...
 
 convertGenericKey ( $key)
 Convert a "generic" reversible cache key into one for this cache. More...
 
 debug ( $text)
 
 getExpirationAsTimestamp ( $exptime)
 Convert an optionally relative timestamp to an absolute time. More...
 
 getExpirationAsTTL ( $exptime)
 Convert an optionally absolute expiry time to a relative time. More...
 
 getSerialized ( $value, $key)
 Get the serialized form a value, using any applicable prepared value. More...
 
 guessSerialValueSize ( $value, $depth=0, &$loops=0)
 Estimate the size of a variable once serialized. More...
 
 isInteger ( $value)
 Check if a value is an integer. More...
 
 isRelativeExpiration ( $exptime)
 
 makeLockKey ( $key)
 
 makeValueOrSegmentList ( $key, $value, $exptime, $flags)
 Determine the entry (inline or segment list) to store under a key to save the value. More...
 
 mergeViaCas ( $key, callable $callback, $exptime, $attempts, $flags)
 
 resolveSegments ( $key, $mainValue)
 Get and reassemble the chunks of blob at the given key. More...
 
 setLastError ( $err)
 Set the "last error" registry. More...
 
 updateOpStats (string $op, array $keyInfo)
 
- Protected Member Functions inherited from BagOStuff
 componentsFromGenericKey ( $key)
 Extract the components from a "generic" reversible cache key. More...
 
 determineKeyPrefixForStats ( $key)
 
 fieldHasFlags ( $field, $flags)
 
 genericKeyFromComponents (... $components)
 At a minimum, there must be a keyspace and collection name component. More...
 
 mergeFlagMaps (array $bags)
 Merge the flag maps of one or more BagOStuff objects into a "lowest common denominator" map. More...
 
 proxyCall ( $method, $arg0Sig, $resSig, array $genericArgs)
 Call a method on behalf of wrapper BagOStuff instance that uses "generic" keys. More...
 

Protected Attributes

Exception[] $connFailureErrors = []
 Map of (shard index => Exception) More...
 
float[] $connFailureTimes = []
 Map of (shard index => UNIX timestamps) More...
 
IMaintainableDatabase[] $conns
 Map of (shard index => DB handle) More...
 
ILoadBalancer null $globalKeyLb
 
int $lastGarbageCollect = 0
 UNIX timestamp. More...
 
ILoadBalancer null $localKeyLb
 
string null $multiPrimaryModeType
 Multi-primary mode DB type ("mysql",...); null if not enabled. More...
 
int $numTableShards = 1
 Number of table shards to use on each server. More...
 
int $purgeLimit = 100
 Max expired rows to purge during randomized garbage collection. More...
 
int $purgePeriod = 10
 Average number of writes required to trigger garbage collection. More...
 
bool $replicaOnly
 Whether to use replicas instead of primaries (if using LoadBalancer) More...
 
array[] $serverInfos = []
 (server index => server config) More...
 
string[] $serverTags = []
 (server index => tag/host name) More...
 
string $tableName = 'objectcache'
 
int $writeBatchSize = 100
 
- Protected Attributes inherited from MediumSpecificBagOStuff
callable[] $busyCallbacks = []
 
int $lastError = self::ERR_NONE
 ERR_* class constant. More...
 
array< string, array > $locks = []
 Map of (key => (class, depth, expiry) More...
 
array[] $preparedValues = []
 Map of (key => (PHP variable value, serialized value)) More...
 
int $segmentationSize
 Bytes; chunk size of segmented cache values. More...
 
int $segmentedValueMaxSize
 Bytes; maximum total size of a segmented cache value. More...
 
int $syncTimeout
 Seconds. More...
 
const LOCK_DEPTH = 1
 
const LOCK_EXPIRY = 3
 
const LOCK_RCLASS = 0
 
const LOCK_TIME = 2
 
const METRIC_OP_ADD = 'add'
 
const METRIC_OP_CAS = 'cas'
 
const METRIC_OP_CHANGE_TTL = 'change_ttl'
 
const METRIC_OP_DECR = 'decr'
 
const METRIC_OP_DELETE = 'delete'
 
const METRIC_OP_GET = 'get'
 
const METRIC_OP_INCR = 'incr'
 
const METRIC_OP_SET = 'set'
 
- Protected Attributes inherited from BagOStuff
callable null $asyncHandler
 
int[] $attrMap = []
 Map of (ATTR_* class constant => QOS_* class constant) More...
 
string $keyspace
 Default keyspace; used by makeKey() More...
 
LoggerInterface $logger
 
StatsdDataFactoryInterface $stats
 
array< string, array > $wrapperInfoByPrefix = []
 Cache key processing callbacks and info for metrics. More...
 

Private Member Functions

 addCasTokenFields (IDatabase $db, array $fields)
 Either append a 'castoken' field or append the fields needed to compute the CAS token. More...
 
 buildExistenceConditions (IDatabase $db, $keys, string $time)
 WHERE conditions that check for existence and liveness of keys. More...
 
 buildIncrUpsertSet (IDatabase $db, int $step, int $init, int $expiry, string $mt, int $mtUnixTs)
 SET array for handling key overwrites when a live or stale key exists. More...
 
 buildUpsertRow (IDatabase $db, $key, $serialValue, int $expiry, string $mt)
 INSERT array for handling key writes/overwrites when no live nor stale key exists. More...
 
 buildUpsertSetForOverwrite (IDatabase $db, $serialValue, int $expiry, string $mt)
 SET array for handling key overwrites when a live or stale key exists. More...
 
 dbDecodeSerialValue (IDatabase $db, $blob)
 
 dbEncodeSerialValue (IDatabase $db, $serialValue)
 
 decodeDbExpiry (IDatabase $db, string $dbExpiry)
 
 deleteServerObjectsExpiringBefore (IDatabase $db, $timestamp, $limit, &$keysDeletedCount=0, array $progress=null)
 
 doIncr ( $key, $value=1, $flags=0)
 
 encodeDbExpiry (IDatabase $db, int $expiry)
 
 fetchBlobs (array $keys, bool $getCasToken=false)
 
 getCasTokenFromRow (IDatabase $db, stdClass $row)
 Get a CAS token from a SELECT result row. More...
 
 getConnection ( $shardIndex)
 Get a connection to the specified database. More...
 
 getConnectionFromServerInfo ( $shardIndex, array $server)
 
 getConnectionViaLoadBalancer ( $shardIndex)
 
 getKeyLocation ( $key)
 Get the server index and table name for a given key. More...
 
 getShardServerIndexes ()
 
 getShardServerIndexForTag (string $tag)
 
 getTableNameByShard ( $index)
 Get the table name for a given shard index. More...
 
 handleDBError (DBError $exception, $shardIndex)
 Handle a DBError which occurred during a read operation. More...
 
 initSqliteDatabase (IMaintainableDatabase $db)
 
 isMultiPrimaryModeEnabled ()
 
 makeNewKeyExpiry ( $exptime, int $nowTsUnix)
 
 makeTimestampedModificationToken ( $mtime, IDatabase $db)
 Make a modtoken column value with the original time and source database server of a write. More...
 
 markServerDown (DBError $exception, $shardIndex)
 Mark a server down due to a DBConnectionError exception. More...
 
 modifyBlobs (callable $tableWriteCallback, float $mtime, array $argsByKey, int $flags, &$resByKey=[])
 
 modifyTableSpecificBlobsForAdd (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
 Insert key/value pairs belonging to a partition table on the the given server. More...
 
 modifyTableSpecificBlobsForCas (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
 Insert key/value pairs belonging to a partition table on the the given server. More...
 
 modifyTableSpecificBlobsForChangeTTL (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
 Update the TTL for keys belonging to a partition table on the the given server. More...
 
 modifyTableSpecificBlobsForDelete (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
 Purge/tombstone key/value pairs belonging to a partition table on the the given server. More...
 
 modifyTableSpecificBlobsForIncrInit (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
 Either increment a counter key, if it exists, or initialize it, otherwise. More...
 
 modifyTableSpecificBlobsForSet (IDatabase $db, string $ptable, float $mtime, array $argsByKey, array &$resByKey)
 Set key/value pairs belonging to a partition table on the the given server. More...
 
 newLockingWriteSectionModificationTimestamp ( $key, &$scope)
 Get a scoped lock and modification timestamp for a critical section of reads/writes. More...
 
 occasionallyGarbageCollect (IDatabase $db)
 
 setAndLogDBError (DBError $exception)
 
 silenceTransactionProfiler ()
 Silence the transaction profiler until the return value falls out of scope. More...
 
 waitForReplication ( $shardIndex)
 Wait for replica DBs to catch up to the primary DB. More...
 

Private Attributes

const BLOB_CASTOKEN = 2
 
const BLOB_EXPIRY = 1
 
const BLOB_VALUE = 0
 
const GC_DELAY_SEC = 1
 How many seconds must pass before triggering a garbage collection. More...
 
const INF_TIMESTAMP_PLACEHOLDER = '99991231235959'
 Placeholder timestamp to use for TTL_INDEFINITE that can be stored in all RDBMs types. More...
 
const SAFE_CLOCK_BOUND_SEC = 15
 A number of seconds well above any expected clock skew. More...
 
const SAFE_PURGE_DELAY_SEC = 3600
 A number of seconds well above any expected clock skew and replication lag. More...
 
const SHARD_GLOBAL = 'global'
 
const SHARD_LOCAL = 'local'
 
const TOMB_EXPTIME = -self::SAFE_CLOCK_BOUND_SEC
 Relative seconds-to-live to use for tombstones. More...
 
const TOMB_SERIAL = ''
 Distinct string for tombstones stored in the "serialized" value column. More...
 

Additional Inherited Members

- Public Attributes inherited from BagOStuff
const READ_LATEST = 1
 Bitfield constants for get()/getMulti(); these are only advisory. More...
 
const READ_VERIFIED = 2
 
const WRITE_ALLOW_SEGMENTS = 16
 
const WRITE_BACKGROUND = 64
 
const WRITE_CACHE_ONLY = 8
 
const WRITE_PRUNE_SEGMENTS = 32
 
const WRITE_SYNC = 4
 Bitfield constants for set()/merge(); these are only advisory. More...
 
- Public Attributes inherited from Wikimedia\LightweightObjectStore\StorageAwareness
const ATTR_DURABILITY = 2
 Durability of writes; see QOS_DURABILITY_* (higher means stronger) More...
 
const ATTR_EMULATION = 1
 Emulation/fallback mode; see QOS_EMULATION_*; higher is better. More...
 
const ERR_NO_RESPONSE = 1
 Storage medium failed to yield a response. More...
 
const ERR_NONE = 0
 No storage medium error. More...
 
const ERR_UNEXPECTED = 3
 Storage medium operation failed due to usage limitations or an I/O error. More...
 
const ERR_UNREACHABLE = 2
 Storage medium could not be reached. More...
 
const QOS_DURABILITY_DISK = 4
 Data is saved to disk and writes do not usually block on fsync() More...
 
const QOS_DURABILITY_NONE = 1
 Data is never saved to begin with (blackhole store) More...
 
const QOS_DURABILITY_RDBMS = 5
 Data is saved to disk and writes usually block on fsync(), like a standard RDBMS. More...
 
const QOS_DURABILITY_SCRIPT = 2
 Data is lost at the end of the current web request or CLI script. More...
 
const QOS_DURABILITY_SERVICE = 3
 Data is lost once the service storing the data restarts. More...
 
const QOS_EMULATION_SQL = 1
 Fallback disk-based SQL store. More...
 
const QOS_UNKNOWN = INF
 Generic "unknown" value; useful for comparisons (always "good enough") More...
 

Detailed Description

RDBMS-based caching module.

The following database sharding schemes are supported:

  • None; all keys map to the same shard
  • Hash; keys map to shards via consistent hashing
  • Keyspace; global keys map to the global shard and non-global keys map to the local shard

The following database replication topologies are supported:

  • A primary database server for each shard, all within one datacenter
  • A co-primary database server for each shard within each datacenter

Definition at line 52 of file SqlBagOStuff.php.

Constructor & Destructor Documentation

◆ __construct()

SqlBagOStuff::__construct (   $params)

Create a new backend instance from configuration.

The database servers must be provided by either the "server" parameter, the "servers" parameter, the "globalKeyLB" parameter, or both the "globalKeyLB"/"localKeyLB" paramters.

Parameters include:

  • server: Server config map for Database::factory() that describes the database to use for all key operations in the current region. This is overriden by "servers".
  • servers: Map of tag strings to server config maps, each for Database::factory(), describing the set of database servers on which to distribute key operations in the current region. Data is distributed among the servers via key hashing based on the server tags. Therefore, each tag represents a shard of the dataset. Tags are useful for failover using cold-standby servers and for managing shards with replica servers in multiple regions (each having different hostnames).
  • localKeyLB: ObjectFactory::getObjectFromSpec array yielding ILoadBalancer. This load balancer is used for local keys, e.g. those using makeKey(). This is overriden by "server" and "servers".
  • globalKeyLB: ObjectFactory::getObjectFromSpec array yielding ILoadBalancer. This load balancer is used for local keys, e.g. those using makeGlobalKey(). This is overriden by "server" and "servers".
  • multiPrimaryMode: Whether the portion of the dataset belonging to each tag/shard is replicated among one or more regions, with one "co-primary" server in each region. Queries are issued in a manner that provides Last-Write-Wins eventual consistency. This option requires the "server" or "servers" options. Only MySQL, with statment based replication (log_bin='ON' and binlog_format='STATEMENT') is supported. Also, the modtoken column must exist on the objectcache table(s). [optional]
  • purgePeriod: The average number of object cache writes in between garbage collection operations, where expired entries are removed from the database. Or in other words, the reciprocal of the probability of purging on any given write. If this is set to zero, purging will never be done. [optional]
  • purgeLimit: Maximum number of rows to purge at once. [optional]
  • tableName: The table name to use, default is "objectcache". [optional]
  • shards: The number of tables to use for data storage on each server. If greater than 1, table names are formed in the style objectcacheNNN where NNN is the shard index, between 0 and shards-1. The number of digits used in the suffix is the minimum number required to hold the largest shard index. Data is distributed among the tables via key hashing. This helps mitigate MySQL bugs 61735 and 61736. [optional]
  • replicaOnly: Whether to only use replica servers and only support read operations. This option requires the use of LoadBalancer ("localKeyLB"/"globalKeyLB") and should only be used by ReplicatedBagOStuff. [optional]
  • syncTimeout: Max seconds to wait for replica DBs to catch up for WRITE_SYNC. [optional]
  • writeBatchSize: Default maximum number of rows to change in each query for write operations that can be chunked into a set of smaller writes. [optional]
Parameters
array$params

Definition at line 159 of file SqlBagOStuff.php.

References $globalKeyLb, $tableName, Wikimedia\LightweightObjectStore\StorageAwareness\ATTR_DURABILITY, Wikimedia\LightweightObjectStore\StorageAwareness\ATTR_EMULATION, Wikimedia\LightweightObjectStore\StorageAwareness\QOS_DURABILITY_RDBMS, and Wikimedia\LightweightObjectStore\StorageAwareness\QOS_EMULATION_SQL.

Member Function Documentation

◆ addCasTokenFields()

SqlBagOStuff::addCasTokenFields ( IDatabase  $db,
array  $fields 
)
private

Either append a 'castoken' field or append the fields needed to compute the CAS token.

Parameters
IDatabase$db
string[]$fieldsSELECT field array
Returns
string[] SELECT field array

Definition at line 1303 of file SqlBagOStuff.php.

References $type, Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\buildConcat(), and Wikimedia\Rdbms\IDatabase\getType().

Referenced by fetchBlobs(), and modifyTableSpecificBlobsForCas().

◆ buildExistenceConditions()

SqlBagOStuff::buildExistenceConditions ( IDatabase  $db,
  $keys,
string  $time 
)
private

WHERE conditions that check for existence and liveness of keys.

Parameters
IDatabase$db
string[] | string$keys
string$timeUNIX modification timestamp with 6 decimal places
Returns
array

Definition at line 1124 of file SqlBagOStuff.php.

References $keys, Wikimedia\Rdbms\IDatabase\addQuotes(), and Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by fetchBlobs(), modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), and modifyTableSpecificBlobsForChangeTTL().

◆ buildIncrUpsertSet()

SqlBagOStuff::buildIncrUpsertSet ( IDatabase  $db,
int  $step,
int  $init,
int  $expiry,
string  $mt,
int  $mtUnixTs 
)
private

SET array for handling key overwrites when a live or stale key exists.

Parameters
IDatabase$db
int$stepPositive counter incrementation value
int$initPositive initial counter value
int$expiryExpiration timestamp or TTL_INDEFINITE
string$mtModification token
int$mtUnixTsUNIX timestamp of modification token
Returns
array

Definition at line 1218 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\buildIntegerCast(), Wikimedia\Rdbms\IDatabase\conditional(), dbEncodeSerialValue(), encodeDbExpiry(), isMultiPrimaryModeEnabled(), and Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by modifyTableSpecificBlobsForIncrInit().

◆ buildUpsertRow()

SqlBagOStuff::buildUpsertRow ( IDatabase  $db,
  $key,
  $serialValue,
int  $expiry,
string  $mt 
)
private

INSERT array for handling key writes/overwrites when no live nor stale key exists.

Parameters
IDatabase$db
string$key
string | int$serialValueNew value
int$expiryExpiration timestamp or TTL_INDEFINITE
string$mtModification token
Returns
array

Definition at line 1142 of file SqlBagOStuff.php.

References dbEncodeSerialValue(), encodeDbExpiry(), and isMultiPrimaryModeEnabled().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), modifyTableSpecificBlobsForIncrInit(), and modifyTableSpecificBlobsForSet().

◆ buildUpsertSetForOverwrite()

SqlBagOStuff::buildUpsertSetForOverwrite ( IDatabase  $db,
  $serialValue,
int  $expiry,
string  $mt 
)
private

SET array for handling key overwrites when a live or stale key exists.

Parameters
IDatabase$db
string | int$serialValueNew value
int$expiryExpiration timestamp or TTL_INDEFINITE
string$mtModification token
Returns
array

Definition at line 1170 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\conditional(), dbEncodeSerialValue(), encodeDbExpiry(), and isMultiPrimaryModeEnabled().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), and modifyTableSpecificBlobsForSet().

◆ createTables()

SqlBagOStuff::createTables ( )

Create the shard tables on all databases (e.g.

via eval.php/shell.php)

Definition at line 1788 of file SqlBagOStuff.php.

References $numTableShards, getConnection(), getShardServerIndexes(), and getTableNameByShard().

◆ dbDecodeSerialValue()

SqlBagOStuff::dbDecodeSerialValue ( IDatabase  $db,
  $blob 
)
private
Parameters
IDatabase$db
Blob | string | int$blob
Returns
string|int

Definition at line 1292 of file SqlBagOStuff.php.

References $blob, Wikimedia\Rdbms\IDatabase\decodeBlob(), and MediumSpecificBagOStuff\isInteger().

Referenced by fetchBlobs(), getCasTokenFromRow(), modifyTableSpecificBlobsForChangeTTL(), and modifyTableSpecificBlobsForIncrInit().

◆ dbEncodeSerialValue()

SqlBagOStuff::dbEncodeSerialValue ( IDatabase  $db,
  $serialValue 
)
private
Parameters
IDatabase$db
string | int$serialValue
Returns
string|int

Definition at line 1283 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\encodeBlob().

Referenced by buildIncrUpsertSet(), buildUpsertRow(), and buildUpsertSetForOverwrite().

◆ decodeDbExpiry()

SqlBagOStuff::decodeDbExpiry ( IDatabase  $db,
string  $dbExpiry 
)
private
Parameters
IDatabase$db
string$dbExpiryDB timestamp of expiration
Returns
int UNIX timestamp of expiration or TTL_INDEFINITE

Definition at line 1272 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by fetchBlobs().

◆ decr()

SqlBagOStuff::decr (   $key,
  $value = 1,
  $flags = 0 
)

Decrease stored value of $key by $value while preserving its TTL.

Parameters
string$key
int$valueValue to subtract from $key (default: 1) [optional]
int$flagsBit field of class WRITE_* constants [optional]
Returns
int|bool New value or false on failure

Reimplemented from BagOStuff.

Definition at line 321 of file SqlBagOStuff.php.

References doIncr().

◆ deleteAll()

SqlBagOStuff::deleteAll ( )

Delete content of shard tables in every server.

Return true if the operation is successful, false otherwise.

Returns
bool

Definition at line 1529 of file SqlBagOStuff.php.

References $numTableShards, getConnection(), getShardServerIndexes(), getTableNameByShard(), handleDBError(), and silenceTransactionProfiler().

◆ deleteObjectsExpiringBefore()

SqlBagOStuff::deleteObjectsExpiringBefore (   $timestamp,
callable  $progress = null,
  $limit = INF,
string  $tag = null 
)

Delete all objects expiring before a certain date.

Parameters
string | int$timestampThe reference date in MW or TS_UNIX format
callable | null$progressOptional, a function which will be called regularly during long-running operations with the percentage progress as the first parameter. [optional]
int$limitMaximum number of keys to delete [default: INF]
string | null$tagTag to purge a single shard only. This is only supported when server tags are used in configuration.
Returns
bool Success; false if unimplemented

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1382 of file SqlBagOStuff.php.

References deleteServerObjectsExpiringBefore(), getConnection(), getShardServerIndexes(), getShardServerIndexForTag(), handleDBError(), and silenceTransactionProfiler().

Referenced by expireAll().

◆ deleteServerObjectsExpiringBefore()

SqlBagOStuff::deleteServerObjectsExpiringBefore ( IDatabase  $db,
  $timestamp,
  $limit,
$keysDeletedCount = 0,
array  $progress = null 
)
private
Parameters
IDatabase$db
string | int$timestamp
int$limitMaximum number of rows to delete in total
int&$keysDeletedCount
null|array{fn:callback,serversDone:int,serversTotal:int}$progress
Exceptions
DBError

Definition at line 1430 of file SqlBagOStuff.php.

References $keys, $numTableShards, $res, Wikimedia\Rdbms\IDatabase\addQuotes(), Wikimedia\Rdbms\IDatabase\affectedRows(), Wikimedia\Rdbms\IDatabase\delete(), BagOStuff\getCurrentTime(), getTableNameByShard(), isMultiPrimaryModeEnabled(), SAFE_PURGE_DELAY_SEC, Wikimedia\Rdbms\IDatabase\select(), and Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by deleteObjectsExpiringBefore(), and occasionallyGarbageCollect().

◆ doAdd()

SqlBagOStuff::doAdd (   $key,
  $value,
  $exptime = 0,
  $flags = 0 
)
protected

Insert an item if it does not already exist.

Parameters
string$key
mixed$value
int$exptime
int$flagsBitfield of BagOStuff::WRITE_* constants (since 1.33)
Returns
bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 259 of file SqlBagOStuff.php.

References modifyBlobs(), and newLockingWriteSectionModificationTimestamp().

◆ doCas()

SqlBagOStuff::doCas (   $casToken,
  $key,
  $value,
  $exptime = 0,
  $flags = 0 
)
protected

Check and set an item.

Parameters
mixed$casToken
string$key
mixed$value
int$exptimeEither an interval in seconds or a unix timestamp for expiry
int$flagsBitfield of BagOStuff::WRITE_* constants
Returns
bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 274 of file SqlBagOStuff.php.

References modifyBlobs(), and newLockingWriteSectionModificationTimestamp().

◆ doChangeTTL()

SqlBagOStuff::doChangeTTL (   $key,
  $exptime,
  $flags 
)
protected
Parameters
string$key
int$exptime
int$flags
Returns
bool

Reimplemented from MediumSpecificBagOStuff.

Definition at line 289 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doChangeTTLMulti()

SqlBagOStuff::doChangeTTLMulti ( array  $keys,
  $exptime,
  $flags = 0 
)
Parameters
string[]$keysList of keys
int$exptimeTTL or UNIX timestamp
int$flagsBitfield of BagOStuff::WRITE_* constants
Returns
bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 408 of file SqlBagOStuff.php.

References $keys, BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doDelete()

SqlBagOStuff::doDelete (   $key,
  $flags = 0 
)
protected

Delete an item.

Parameters
string$key
int$flagsBitfield of BagOStuff::WRITE_* constants
Returns
bool True if the item was deleted or not found, false on failure

Reimplemented from MediumSpecificBagOStuff.

Definition at line 248 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doDeleteMulti()

SqlBagOStuff::doDeleteMulti ( array  $keys,
  $flags = 0 
)
protected
Parameters
string[]$keysList of keys
int$flagsBitfield of BagOStuff::WRITE_* constants
Returns
bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 397 of file SqlBagOStuff.php.

References $keys, BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doGet()

SqlBagOStuff::doGet (   $key,
  $flags = 0,
$casToken = null 
)
protected
Parameters
string$key
int$flagsBitfield of BagOStuff::READ_* constants [optional]
mixed | null&$casTokencas() token if MediumSpecificBagOStuff::PASS_BY_REF [returned]
Returns
mixed Returns false on failure or if the item does not exist

Reimplemented from MediumSpecificBagOStuff.

Definition at line 216 of file SqlBagOStuff.php.

References BLOB_CASTOKEN, fetchBlobs(), unserialize(), and MediumSpecificBagOStuff\updateOpStats().

◆ doGetMulti()

SqlBagOStuff::doGetMulti ( array  $keys,
  $flags = 0 
)
protected

Get an associative array containing the item for each of the keys that have items.

Parameters
string[]$keysList of keys
int$flagsBitfield; supports READ_LATEST [optional]
Returns
array Map of (key => value) for existing keys; preserves the order of $keys

Reimplemented from MediumSpecificBagOStuff.

Definition at line 356 of file SqlBagOStuff.php.

References $keys, BLOB_VALUE, fetchBlobs(), unserialize(), and MediumSpecificBagOStuff\updateOpStats().

◆ doIncr()

SqlBagOStuff::doIncr (   $key,
  $value = 1,
  $flags = 0 
)
private

◆ doLock()

SqlBagOStuff::doLock (   $key,
  $timeout = 6,
  $exptime = 6 
)
See also
MediumSpecificBagOStuff::lock()
Parameters
string$key
int$timeoutLock wait timeout; 0 for non-blocking [optional]
int$exptimeLock time-to-live 1 day maximum [optional]
Returns
float|null UNIX timestamp of acquisition; null on failure

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1547 of file SqlBagOStuff.php.

References getConnection(), getKeyLocation(), handleDBError(), and silenceTransactionProfiler().

◆ doSet()

SqlBagOStuff::doSet (   $key,
  $value,
  $exptime = 0,
  $flags = 0 
)
protected

Set an item.

Parameters
string$key
mixed$value
int$exptimeEither an interval in seconds or a unix timestamp for expiry
int$flagsBitfield of BagOStuff::WRITE_* constants
Returns
bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 237 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doSetMulti()

SqlBagOStuff::doSetMulti ( array  $data,
  $exptime = 0,
  $flags = 0 
)
protected
Parameters
mixed[]$dataMap of (key => value)
int$exptimeEither an interval in seconds or a unix timestamp for expiry
int$flagsBitfield of BagOStuff::WRITE_* constants
Returns
bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 381 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ doUnlock()

SqlBagOStuff::doUnlock (   $key)
See also
MediumSpecificBagOStuff::unlock()
Parameters
string$key
Returns
bool Success

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1568 of file SqlBagOStuff.php.

References getConnection(), getKeyLocation(), handleDBError(), and silenceTransactionProfiler().

◆ encodeDbExpiry()

SqlBagOStuff::encodeDbExpiry ( IDatabase  $db,
int  $expiry 
)
private
Parameters
IDatabase$db
int$expiryUNIX timestamp of expiration or TTL_INDEFINITE
Returns
string

Definition at line 1259 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\timestamp().

Referenced by buildIncrUpsertSet(), buildUpsertRow(), buildUpsertSetForOverwrite(), and modifyTableSpecificBlobsForChangeTTL().

◆ expireAll()

SqlBagOStuff::expireAll ( )

Definition at line 1378 of file SqlBagOStuff.php.

References deleteObjectsExpiringBefore(), and BagOStuff\getCurrentTime().

◆ fetchBlobs()

SqlBagOStuff::fetchBlobs ( array  $keys,
bool  $getCasToken = false 
)
private
Parameters
string[]$keys
bool$getCasTokenWhether to get a CAS token
Returns
array<string,array|null> Order-preserved map of (key => (value,expiry,token) or null)

Definition at line 507 of file SqlBagOStuff.php.

References $keys, $res, addCasTokenFields(), buildExistenceConditions(), dbDecodeSerialValue(), MediumSpecificBagOStuff\debug(), decodeDbExpiry(), getCasTokenFromRow(), getConnection(), BagOStuff\getCurrentTime(), getKeyLocation(), handleDBError(), and silenceTransactionProfiler().

Referenced by doGet(), doGetMulti(), and doIncr().

◆ getCasTokenFromRow()

SqlBagOStuff::getCasTokenFromRow ( IDatabase  $db,
stdClass  $row 
)
private

Get a CAS token from a SELECT result row.

Parameters
IDatabase$db
stdClass$rowA row for a key
Returns
string CAS token

Definition at line 1337 of file SqlBagOStuff.php.

References dbDecodeSerialValue().

Referenced by fetchBlobs(), and modifyTableSpecificBlobsForCas().

◆ getConnection()

SqlBagOStuff::getConnection (   $shardIndex)
private

Get a connection to the specified database.

Parameters
int | string$shardIndexServer index or self::SHARD_LOCAL/self::SHARD_GLOBAL
Returns
IMaintainableDatabase
Exceptions
DBConnectionError
UnexpectedValueException

Definition at line 427 of file SqlBagOStuff.php.

References getConnectionFromServerInfo(), getConnectionViaLoadBalancer(), and BagOStuff\getCurrentTime().

Referenced by createTables(), deleteAll(), deleteObjectsExpiringBefore(), doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ getConnectionFromServerInfo()

SqlBagOStuff::getConnectionFromServerInfo (   $shardIndex,
array  $server 
)
private
Parameters
int$shardIndex
array$serverServer config map
Returns
IMaintainableDatabase
Exceptions
DBError

Definition at line 1676 of file SqlBagOStuff.php.

References DBO_TRX, Wikimedia\Rdbms\IDatabase\getType(), initSqliteDatabase(), and Wikimedia\Rdbms\IDatabase\tableExists().

Referenced by getConnection().

◆ getConnectionViaLoadBalancer()

SqlBagOStuff::getConnectionViaLoadBalancer (   $shardIndex)
private
Parameters
string$shardIndexself::SHARD_LOCAL/self::SHARD_GLOBAL
Returns
IMaintainableDatabase
Exceptions
DBError

Definition at line 1652 of file SqlBagOStuff.php.

References DB_PRIMARY, DB_REPLICA, and SHARD_LOCAL.

Referenced by getConnection().

◆ getKeyLocation()

SqlBagOStuff::getKeyLocation (   $key)
private

Get the server index and table name for a given key.

Parameters
string$key
Returns
array (server index or self::SHARD_LOCAL/self::SHARD_GLOBAL, table name)

Definition at line 459 of file SqlBagOStuff.php.

References $globalKeyLb, $numTableShards, $serverTags, ArrayUtils\consistentHashSort(), and getTableNameByShard().

Referenced by doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ getShardServerIndexes()

SqlBagOStuff::getShardServerIndexes ( )
private
Returns
string[]|int[] List of server indexes or self::SHARD_LOCAL/self::SHARD_GLOBAL

Definition at line 1804 of file SqlBagOStuff.php.

References SHARD_GLOBAL, and SHARD_LOCAL.

Referenced by createTables(), deleteAll(), and deleteObjectsExpiringBefore().

◆ getShardServerIndexForTag()

SqlBagOStuff::getShardServerIndexForTag ( string  $tag)
private
Parameters
string$tag
Returns
int Server index for use with ::getConnection()
Exceptions
InvalidArgumentExceptionIf tag is unknown

Definition at line 1827 of file SqlBagOStuff.php.

Referenced by deleteObjectsExpiringBefore().

◆ getTableNameByShard()

SqlBagOStuff::getTableNameByShard (   $index)
private

Get the table name for a given shard index.

Parameters
int | null$index
Returns
string

Definition at line 492 of file SqlBagOStuff.php.

References $tableName.

Referenced by createTables(), deleteAll(), deleteServerObjectsExpiringBefore(), and getKeyLocation().

◆ handleDBError()

SqlBagOStuff::handleDBError ( DBError  $exception,
  $shardIndex 
)
private

Handle a DBError which occurred during a read operation.

Parameters
DBError$exception
int | string$shardIndexServer index or self::SHARD_LOCAL/self::SHARD_GLOBAL

Definition at line 1709 of file SqlBagOStuff.php.

References markServerDown(), and setAndLogDBError().

Referenced by deleteAll(), deleteObjectsExpiringBefore(), doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ incr()

SqlBagOStuff::incr (   $key,
  $value = 1,
  $flags = 0 
)

Increase stored value of $key by $value while preserving its TTL.

Parameters
string$keyKey to increase
int$valueValue to add to $key (default: 1) [optional]
int$flagsBit field of class WRITE_* constants [optional]
Returns
int|bool New value or false on failure

Reimplemented from BagOStuff.

Definition at line 317 of file SqlBagOStuff.php.

References doIncr().

◆ incrWithInit()

SqlBagOStuff::incrWithInit (   $key,
  $exptime,
  $value = 1,
  $init = null,
  $flags = 0 
)

Increase the value of the given key (no TTL change) if it exists or create it otherwise.

This will create the key with the value $init and TTL $exptime instead if not present. Callers should make sure that both ($init - $value) and $exptime are invariants for all operations to any given key. The value of $init should be at least that of $value.

Parameters
string$keyKey built via makeKey() or makeGlobalKey()
int$exptimeTime-to-live (in seconds) or a UNIX timestamp expiration
int$valueAmount to increase the key value by [default: 1]
int | null$initValue to initialize the key to if it does not exist [default: $value]
int$flagsBit field of class WRITE_* constants [optional]
Returns
int|bool New value or false on failure
Since
1.24

Reimplemented from MediumSpecificBagOStuff.

Definition at line 300 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime(), and modifyBlobs().

◆ initSqliteDatabase()

◆ isMultiPrimaryModeEnabled()

SqlBagOStuff::isMultiPrimaryModeEnabled ( )
private

◆ makeKeyInternal()

SqlBagOStuff::makeKeyInternal (   $keyspace,
  $components 
)

Make a cache key for the given keyspace and components.

Long components might be converted to respective hashes due to size constraints. In extreme cases, all of them might be combined into a single hash component.

Access: internal
This method should not be used outside of BagOStuff (since 1.36)
Parameters
string$keyspaceKeyspace component
string[] | int[]$componentsKey components (key collection name first)
Returns
string Keyspace-prepended list of encoded components as a colon-separated value
Since
1.27

Reimplemented from BagOStuff.

Definition at line 1585 of file SqlBagOStuff.php.

References BagOStuff\$keyspace.

◆ makeNewKeyExpiry()

SqlBagOStuff::makeNewKeyExpiry (   $exptime,
int  $nowTsUnix 
)
private
Parameters
int$exptimeRelative or absolute expiration
int$nowTsUnixCurrent UNIX timestamp
Returns
int UNIX timestamp or TTL_INDEFINITE

Definition at line 1032 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\getExpirationAsTimestamp().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), modifyTableSpecificBlobsForIncrInit(), and modifyTableSpecificBlobsForSet().

◆ makeTimestampedModificationToken()

SqlBagOStuff::makeTimestampedModificationToken (   $mtime,
IDatabase  $db 
)
private

Make a modtoken column value with the original time and source database server of a write.

Parameters
int | float$mtimeUNIX modification timestamp
IDatabase$dbHandle to the primary database server sourcing the write
Returns
string String of the form "<SECONDS_SOURCE><MICROSECONDS>", where SECONDS_SOURCE is "<35 bit seconds portion of UNIX time><32 bit database server ID>" as 13 base 36 chars, and MICROSECONDS is "<20 bit microseconds portion of UNIX time>" as 4 base 36 chars

Definition at line 1083 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\getServerName(), and Wikimedia\Rdbms\IDatabase\getTopologyBasedServerId().

Referenced by modifyTableSpecificBlobsForAdd(), modifyTableSpecificBlobsForCas(), modifyTableSpecificBlobsForChangeTTL(), modifyTableSpecificBlobsForDelete(), modifyTableSpecificBlobsForIncrInit(), and modifyTableSpecificBlobsForSet().

◆ markServerDown()

SqlBagOStuff::markServerDown ( DBError  $exception,
  $shardIndex 
)
private

Mark a server down due to a DBConnectionError exception.

Parameters
DBError$exception
int | string$shardIndexServer index or self::SHARD_LOCAL/self::SHARD_GLOBAL

Definition at line 1737 of file SqlBagOStuff.php.

References BagOStuff\getCurrentTime().

Referenced by handleDBError().

◆ modifyBlobs()

SqlBagOStuff::modifyBlobs ( callable  $tableWriteCallback,
float  $mtime,
array  $argsByKey,
int  $flags,
$resByKey = [] 
)
private
Parameters
callable$tableWriteCallbackCallback the takes the following arguments:
  • IDatabase instance
  • Partition table name string
  • UNIX modification timestamp
  • Map of (key => list of arguments) for keys belonging to the server/table partition
  • Map of (key => result) [returned]
float$mtimeUNIX modification timestamp
array<string,array>$argsByKey Map of (key => list of arguments)
int$flagsBitfield of BagOStuff::WRITE_* constants
array<string,mixed>&$resByKey Order-preserved map of (key => result) [returned]
Returns
bool Whether all keys were processed
Parameters
-taint$argsByKeynone

Definition at line 582 of file SqlBagOStuff.php.

References $args, $success, BagOStuff\fieldHasFlags(), getConnection(), getKeyLocation(), handleDBError(), occasionallyGarbageCollect(), silenceTransactionProfiler(), and waitForReplication().

Referenced by doAdd(), doCas(), doChangeTTL(), doChangeTTLMulti(), doDelete(), doDeleteMulti(), doIncr(), doSet(), doSetMulti(), and incrWithInit().

◆ modifyTableSpecificBlobsForAdd()

SqlBagOStuff::modifyTableSpecificBlobsForAdd ( IDatabase  $db,
string  $ptable,
float  $mtime,
array  $argsByKey,
array &  $resByKey 
)
private

Insert key/value pairs belonging to a partition table on the the given server.

If the current row for a key exists and has an integral UNIX timestamp of expiration greater than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Aquisition of advisory key locks must be handled by calling functions.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp.

Parameters
IDatabase$dbHandle to the database server where the argument keys belong
string$ptableName of the partition table where the argument keys belong
float$mtimeUNIX modification timestamp
array<string,array>$argsByKey Non-empty (key => (value,exptime)) map
array<string,mixed>&$resByKey Map of (key => result) prefilled with false [returned]
Exceptions
DBError

Definition at line 768 of file SqlBagOStuff.php.

References buildExistenceConditions(), buildUpsertRow(), buildUpsertSetForOverwrite(), MediumSpecificBagOStuff\getSerialized(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\selectFieldValues(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForCas()

SqlBagOStuff::modifyTableSpecificBlobsForCas ( IDatabase  $db,
string  $ptable,
float  $mtime,
array  $argsByKey,
array &  $resByKey 
)
private

Insert key/value pairs belonging to a partition table on the the given server.

If the current row for a key exists, has an integral UNIX timestamp of expiration greater than that of the provided modification timestamp, and the CAS token does not match, then the write to that key will be aborted with a "false" result. Aquisition of advisory key locks must be handled by calling functions.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp.

Parameters
IDatabase$dbHandle to the database server where the argument keys belong
string$ptableName of the partition table where the argument keys belong
float$mtimeUNIX modification timestamp
array<string,array>$argsByKey Non-empty (key => (value, exptime, CAS token)) map
array<string,mixed>&$resByKey Map of (key => result) prefilled with false [returned]
Exceptions
DBError

Definition at line 832 of file SqlBagOStuff.php.

References $res, addCasTokenFields(), buildExistenceConditions(), buildUpsertRow(), buildUpsertSetForOverwrite(), getCasTokenFromRow(), MediumSpecificBagOStuff\getSerialized(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\select(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForChangeTTL()

SqlBagOStuff::modifyTableSpecificBlobsForChangeTTL ( IDatabase  $db,
string  $ptable,
float  $mtime,
array  $argsByKey,
array &  $resByKey 
)
private

Update the TTL for keys belonging to a partition table on the the given server.

If no current row for a key exists or the current row has an integral UNIX timestamp of expiration less than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp.

Parameters
IDatabase$dbHandle to the database server where the argument keys belong
string$ptableName of the partition table where the argument keys belong
float$mtimeUNIX modification timestamp
array<string,array>$argsByKey Non-empty (key => (exptime)) map
array<string,mixed>&$resByKey Map of (key => result) prefilled with false [returned]
Exceptions
DBError

Definition at line 905 of file SqlBagOStuff.php.

References $args, $res, Wikimedia\Rdbms\IDatabase\affectedRows(), buildExistenceConditions(), buildUpsertRow(), buildUpsertSetForOverwrite(), dbDecodeSerialValue(), encodeDbExpiry(), isMultiPrimaryModeEnabled(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\select(), Wikimedia\Rdbms\IDatabase\update(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForDelete()

SqlBagOStuff::modifyTableSpecificBlobsForDelete ( IDatabase  $db,
string  $ptable,
float  $mtime,
array  $argsByKey,
array &  $resByKey 
)
private

Purge/tombstone key/value pairs belonging to a partition table on the the given server.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token/timestamp, an empty value, and an expiration timestamp dated slightly before the new modification timestamp.

Parameters
IDatabase$dbHandle to the database server where the argument keys belong
string$ptableName of the partition table where the argument keys belong
float$mtimeUNIX modification timestamp
array<string,array>$argsByKey Non-empty (key => []) map
array<string,mixed>&$resByKey Map of (key => result) prefilled with false [returned]
Exceptions
DBError

Definition at line 714 of file SqlBagOStuff.php.

References buildUpsertRow(), buildUpsertSetForOverwrite(), Wikimedia\Rdbms\IDatabase\delete(), isMultiPrimaryModeEnabled(), makeNewKeyExpiry(), makeTimestampedModificationToken(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForIncrInit()

SqlBagOStuff::modifyTableSpecificBlobsForIncrInit ( IDatabase  $db,
string  $ptable,
float  $mtime,
array  $argsByKey,
array &  $resByKey 
)
private

Either increment a counter key, if it exists, or initialize it, otherwise.

If no current row for a key exists or the current row has an integral UNIX timestamp of expiration less than that of the provided modification timestamp, then the key row will be set to the initial value. Otherwise, the current row will be incremented.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully initialized key rows will be assigned a new modification token/timestamp.

Parameters
IDatabase$dbHandle to the database server where the argument keys belong
string$ptableName of the partition table where the argument keys belong
float$mtimeUNIX modification timestamp
array<string,array>$argsByKey Non-empty (key => (step, init, exptime) map
array<string,mixed>&$resByKey Map of (key => result) prefilled with false [returned]
Exceptions
DBError

Definition at line 983 of file SqlBagOStuff.php.

References Wikimedia\Rdbms\IDatabase\affectedRows(), buildIncrUpsertSet(), buildUpsertRow(), dbDecodeSerialValue(), Wikimedia\Rdbms\IDatabase\endAtomic(), MediumSpecificBagOStuff\isInteger(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\selectRow(), Wikimedia\Rdbms\IDatabase\startAtomic(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ modifyTableSpecificBlobsForSet()

SqlBagOStuff::modifyTableSpecificBlobsForSet ( IDatabase  $db,
string  $ptable,
float  $mtime,
array  $argsByKey,
array &  $resByKey 
)
private

Set key/value pairs belonging to a partition table on the the given server.

In multi-primary mode, if the current row for a key exists and has a modification token with a greater integral UNIX timestamp than that of the provided modification timestamp, then the write to that key will be aborted with a "false" result. Successfully modified key rows will be assigned a new modification token using the provided timestamp.

Parameters
IDatabase$dbHandle to the database server where the argument keys belong
string$ptableName of the partition table where the argument keys belong
float$mtimeUNIX modification timestamp
array<string,array>$argsByKey Non-empty (key => (value,exptime)) map
array<string,mixed>&$resByKey Map of (key => result) for succesful writes [returned]
Exceptions
DBError

Definition at line 652 of file SqlBagOStuff.php.

References buildUpsertRow(), buildUpsertSetForOverwrite(), MediumSpecificBagOStuff\getSerialized(), makeNewKeyExpiry(), makeTimestampedModificationToken(), Wikimedia\Rdbms\IDatabase\replace(), MediumSpecificBagOStuff\updateOpStats(), and Wikimedia\Rdbms\IDatabase\upsert().

◆ newLockingWriteSectionModificationTimestamp()

SqlBagOStuff::newLockingWriteSectionModificationTimestamp (   $key,
$scope 
)
private

Get a scoped lock and modification timestamp for a critical section of reads/writes.

This is used instead of BagOStuff::getCurrentTime() for certain writes (such as "add", "incr", and "cas"), for which we want to support tight race conditions where the same key is repeatedly written to by multiple web servers that each get to see the previous value, act on it, and modify it in some way.

It is assumed that this method is normally only invoked from the primary datacenter. A lock is acquired on the primary server of the local datacenter in order to avoid race conditions within the critical section. The clock on the SQL server is used to get the modification timestamp in order to minimize issues with clock drift between web servers; thus key writes will not be rejected due to some web servers having lagged clocks.

Parameters
string$key
?ScopedCallback&$scope Unlocker callback; null on failure [returned]
Returns
string|null UNIX timestamp with 6 decimal places; null on failure

Definition at line 1062 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\lock(), and MediumSpecificBagOStuff\unlock().

Referenced by doAdd(), doCas(), and doIncr().

◆ occasionallyGarbageCollect()

SqlBagOStuff::occasionallyGarbageCollect ( IDatabase  $db)
private
Parameters
IDatabase$db
Exceptions
DBError

Definition at line 1352 of file SqlBagOStuff.php.

References BagOStuff\$asyncHandler, deleteServerObjectsExpiringBefore(), and BagOStuff\getCurrentTime().

Referenced by modifyBlobs().

◆ serialize()

SqlBagOStuff::serialize (   $value)
protected
Parameters
mixed$value
Returns
string|int|false String/integer representation
Note
Special handling is usually needed for integers so incr()/decr() work

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1611 of file SqlBagOStuff.php.

◆ setAndLogDBError()

SqlBagOStuff::setAndLogDBError ( DBError  $exception)
private
Parameters
DBError$exception

Definition at line 1720 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\setLastError().

Referenced by handleDBError(), and waitForReplication().

◆ silenceTransactionProfiler()

SqlBagOStuff::silenceTransactionProfiler ( )
private

Silence the transaction profiler until the return value falls out of scope.

Returns
ScopedCallback|null

Definition at line 1890 of file SqlBagOStuff.php.

References Profiler\instance().

Referenced by deleteAll(), deleteObjectsExpiringBefore(), doLock(), doUnlock(), fetchBlobs(), and modifyBlobs().

◆ unserialize()

SqlBagOStuff::unserialize (   $value)
protected
Parameters
string | int | false$value
Returns
mixed Original value or false on error
Note
Special handling is usually needed for integers so incr()/decr() work

Reimplemented from MediumSpecificBagOStuff.

Definition at line 1625 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\isInteger().

Referenced by doGet(), and doGetMulti().

◆ waitForReplication()

SqlBagOStuff::waitForReplication (   $shardIndex)
private

Wait for replica DBs to catch up to the primary DB.

Parameters
int | string$shardIndexServer index or self::SHARD_LOCAL/self::SHARD_GLOBAL
Returns
bool Success

Definition at line 1852 of file SqlBagOStuff.php.

References MediumSpecificBagOStuff\$busyCallbacks, MediumSpecificBagOStuff\$syncTimeout, setAndLogDBError(), and SHARD_LOCAL.

Referenced by modifyBlobs().

Member Data Documentation

◆ $connFailureErrors

Exception [] SqlBagOStuff::$connFailureErrors = []
protected

Map of (shard index => Exception)

Definition at line 84 of file SqlBagOStuff.php.

◆ $connFailureTimes

float [] SqlBagOStuff::$connFailureTimes = []
protected

Map of (shard index => UNIX timestamps)

Definition at line 82 of file SqlBagOStuff.php.

◆ $conns

IMaintainableDatabase [] SqlBagOStuff::$conns
protected

Map of (shard index => DB handle)

Definition at line 80 of file SqlBagOStuff.php.

◆ $globalKeyLb

ILoadBalancer null SqlBagOStuff::$globalKeyLb
protected

Definition at line 56 of file SqlBagOStuff.php.

Referenced by __construct(), and getKeyLocation().

◆ $lastGarbageCollect

int SqlBagOStuff::$lastGarbageCollect = 0
protected

UNIX timestamp.

Definition at line 63 of file SqlBagOStuff.php.

◆ $localKeyLb

ILoadBalancer null SqlBagOStuff::$localKeyLb
protected

Definition at line 54 of file SqlBagOStuff.php.

◆ $multiPrimaryModeType

string null SqlBagOStuff::$multiPrimaryModeType
protected

Multi-primary mode DB type ("mysql",...); null if not enabled.

Definition at line 77 of file SqlBagOStuff.php.

◆ $numTableShards

int SqlBagOStuff::$numTableShards = 1
protected

Number of table shards to use on each server.

Definition at line 69 of file SqlBagOStuff.php.

Referenced by createTables(), deleteAll(), deleteServerObjectsExpiringBefore(), and getKeyLocation().

◆ $purgeLimit

int SqlBagOStuff::$purgeLimit = 100
protected

Max expired rows to purge during randomized garbage collection.

Definition at line 67 of file SqlBagOStuff.php.

◆ $purgePeriod

int SqlBagOStuff::$purgePeriod = 10
protected

Average number of writes required to trigger garbage collection.

Definition at line 65 of file SqlBagOStuff.php.

◆ $replicaOnly

bool SqlBagOStuff::$replicaOnly
protected

Whether to use replicas instead of primaries (if using LoadBalancer)

Definition at line 75 of file SqlBagOStuff.php.

◆ $serverInfos

array [] SqlBagOStuff::$serverInfos = []
protected

(server index => server config)

Definition at line 59 of file SqlBagOStuff.php.

◆ $serverTags

string [] SqlBagOStuff::$serverTags = []
protected

(server index => tag/host name)

Definition at line 61 of file SqlBagOStuff.php.

Referenced by getKeyLocation().

◆ $tableName

string SqlBagOStuff::$tableName = 'objectcache'
protected

Definition at line 73 of file SqlBagOStuff.php.

Referenced by __construct(), and getTableNameByShard().

◆ $writeBatchSize

int SqlBagOStuff::$writeBatchSize = 100
protected

Definition at line 71 of file SqlBagOStuff.php.

◆ BLOB_CASTOKEN

const SqlBagOStuff::BLOB_CASTOKEN = 2
private

Definition at line 102 of file SqlBagOStuff.php.

Referenced by doGet().

◆ BLOB_EXPIRY

const SqlBagOStuff::BLOB_EXPIRY = 1
private

Definition at line 101 of file SqlBagOStuff.php.

◆ BLOB_VALUE

const SqlBagOStuff::BLOB_VALUE = 0
private

Definition at line 100 of file SqlBagOStuff.php.

Referenced by doGetMulti(), and doIncr().

◆ GC_DELAY_SEC

const SqlBagOStuff::GC_DELAY_SEC = 1
private

How many seconds must pass before triggering a garbage collection.

Definition at line 98 of file SqlBagOStuff.php.

◆ INF_TIMESTAMP_PLACEHOLDER

const SqlBagOStuff::INF_TIMESTAMP_PLACEHOLDER = '99991231235959'
private

Placeholder timestamp to use for TTL_INDEFINITE that can be stored in all RDBMs types.

We use BINARY(14) for MySQL, BLOB for Sqlite, and TIMESTAMPZ for Postgres (which goes up to 294276 AD). The last second of the year 9999 can be stored in all these cases. https://www.postgresql.org/docs/9.0/datatype-datetime.html

Definition at line 110 of file SqlBagOStuff.php.

◆ SAFE_CLOCK_BOUND_SEC

const SqlBagOStuff::SAFE_CLOCK_BOUND_SEC = 15
private

A number of seconds well above any expected clock skew.

Definition at line 90 of file SqlBagOStuff.php.

◆ SAFE_PURGE_DELAY_SEC

const SqlBagOStuff::SAFE_PURGE_DELAY_SEC = 3600
private

A number of seconds well above any expected clock skew and replication lag.

Definition at line 92 of file SqlBagOStuff.php.

Referenced by deleteServerObjectsExpiringBefore().

◆ SHARD_GLOBAL

const SqlBagOStuff::SHARD_GLOBAL = 'global'
private

Definition at line 87 of file SqlBagOStuff.php.

Referenced by getShardServerIndexes().

◆ SHARD_LOCAL

const SqlBagOStuff::SHARD_LOCAL = 'local'
private

◆ TOMB_EXPTIME

const SqlBagOStuff::TOMB_EXPTIME = -self::SAFE_CLOCK_BOUND_SEC
private

Relative seconds-to-live to use for tombstones.

Definition at line 96 of file SqlBagOStuff.php.

◆ TOMB_SERIAL

const SqlBagOStuff::TOMB_SERIAL = ''
private

Distinct string for tombstones stored in the "serialized" value column.

Definition at line 94 of file SqlBagOStuff.php.


The documentation for this class was generated from the following file: