Provide a given client with protection against visible database lag. More...

Inheritance diagram for Wikimedia\Rdbms\ChronologyProtector:

Collaboration diagram for Wikimedia\Rdbms\ChronologyProtector:

[legend]

Public Member Functions
	__construct (BagOStuff $store, array $client, ?int $clientPosIndex, string $secret='')

	applySessionReplicationPosition (ILoadBalancer $lb)
	Apply client "session consistency" replication position to a new ILoadBalancer. More...

	getClientId ()

	getTouched (ILoadBalancer $lb)
	Get the UNIX timestamp when the client last touched the DB, if they did so recently. More...

	persistSessionReplicationPositions (&$clientPosIndex=null)
	Persist any staged client "session consistency" replication positions. More...

	setEnabled ( $enabled)

	setLogger (LoggerInterface $logger)

	setMockTime (&$time)

	setWaitEnabled ( $enabled)

	stageSessionReplicationPosition (ILoadBalancer $lb)
	Update client "session consistency" replication position for an end-of-life ILoadBalancer. More...

Public Attributes
const	POSITION_COOKIE_TTL = 10
	Seconds to store position write index cookies (safely less than POSITION_STORE_TTL) More...

Protected Member Functions
	getCurrentTime ()

	getStartupSessionPositions ()

	getStartupSessionTimestamps ()

	lazyStartup ()
	Load the stored replication positions and touch timestamps for the client. More...

	mergePositions ( $storedValue, array $shutdownPositions, array $shutdownTimestamps, ?int &$clientPosIndex=null)
	Merge the new replication positions with the currently stored ones (highest wins) More...

Protected Attributes
string	$clientId
	Hash of client parameters. More...

string[]	$clientLogInfo
	Map of client information fields for logging. More...

bool	$enabled = true
	Whether reading/writing session consistency replication positions is enabled. More...

string	$key
	Storage key name. More...

LoggerInterface	$logger

bool	$positionWaitsEnabled = true
	Whether waiting on DB servers to reach replication positions is enabled. More...

array< string, DBMasterPos >	$shutdownPositionsByMaster = []
	Map of (DB master name => position) More...

array< string, float >	$shutdownTimestampsByCluster = []
	Map of (DB cluster name => UNIX timestamp) More...

array< string, DBMasterPos >	$startupPositionsByMaster = []
	Map of (DB master name => position) More...

float null	$startupTimestamp
	UNIX timestamp when the client data was loaded. More...

array< string, float >	$startupTimestampsByCluster = []
	Map of (DB cluster name => UNIX timestamp) More...

BagOStuff	$store

int null	$waitForPosIndex
	Expected minimum index of the last write to the position store. More...

Private Attributes
float null	$wallClockOverride

const	FLD_POSITIONS = 'positions'

const	FLD_TIMESTAMPS = 'timestamps'

const	FLD_WRITE_INDEX = 'writeIndex'

const	LOCK_TIMEOUT = 3
	Lock timeout to use for key updates. More...

const	LOCK_TTL = 6
	Lock expiry to use for key updates. More...

const	POSITION_INDEX_WAIT_TIMEOUT = 5
	Max seconds to wait for positions write indexes to appear (e.g. More...

const	POSITION_STORE_TTL = 60
	Seconds to store replication positions. More...

Detailed Description

Provide a given client with protection against visible database lag.

In a nut shell

This class tries to hide visible effects of database lag. It does this by temporarily remembering the database positions after a client makes a write, and on their next web request we will prefer non-lagged database replicas. When replica connections are establshed, we wait up to a few seconds for sufficient replication to have occurred, if they were not yet caught up to that same point.

This ensures a consistent ordering of events as seen by a client. Kind of like Hawking's Chronology Protection Agency.

Purpose

For performance and scalability reasons, almost all data is queried from replica databases. Only queries relating to writing data, are sent to a database master. When rendering a web page with content or activity feeds on it, the very latest information may thus not yet be there. That's okay in general, but if, for example, a client recently changed their preferences or submitted new data, we do our best to make sure their next web response does reflect at least their own recent changes.

How

To explain how it works, we will look at an example lifecycle for a client.

A client is browsing the site. Their web requests are generally read-only and display data from database replicas, which may be a few seconds out of date if a client elsewhere in the world recently modified that same data. If the application is run from multiple data centers, then these web requests may be served from the nearest secondary DC.

A client performs a POST request, perhaps to publish an edit or change their preferences. This request is routed to the primary DC (this is the responsibility of infrastructure outside the web app). There, the data is saved to the database master, after which the database host will asynchronously replicate this to its replicas in the same and any other DCs.

Toward the end of the response to this POST request, the application takes note of the database master's current "position", and save this under a "clientId" key in the ChronologyProtector store. The web response will also set two cookies that are similarly short-lived (about ten seconds): UseDC=master and cpPosIndex=<posIndex>@<write time>#<clientId>.

The ten seconds window is meant to account for the time needed for the database writes to have replicated across all active database replicas, including the cross-dc latency for those further away in any secondary DCs. The "clientId" is placed in the cookie to handle the case where the client IP addresses frequently changes between web requests.

Future web requests from the client should fall in one of two categories:

Within the ten second window. Their UseDC cookie will make them return to the primary DC where we access the ChronologyProtector store and use the database "position" to decide which local database replica to use and on-demand wait a split second for replication to catch up if needed.
After the ten second window. They will be routed to the nearest and possibly different DC. Any local ChronologyProtector store existing there will not be interacted with. A random database replica may be used as the client's own writes are expected to have been applied here by now.

Storage requirements

The store used by ChronologyProtector, as configured via $wgChronologyProtectorStash, should meet the following requirements:

Low latencies. Nearly all web requests that involve a database connection will unconditionally query this store first. It is expected to respond within the order of one millisecond.
Best effort persistence, without active eviction pressure. Data stored here cannot be obtained elsewhere or recomputed. As such, under normal operating conditions, this store should not be full, and should not evict values before their intended expiry time ellapsed.
No replication, local consistency. Each DC may have a fully independent dc-local store associated with ChronologyProtector (no replication across DCs is needed). Local writes must be immediately reflected in subsequent local reads. No intra-dc read lag is allowed.
No redundancy, fast failure. Loss of data will likely be noticable and disruptive to clients, but the data is not considered essential. Under maintenance or unprecedented load, it is recommended to lose some data, instead of compromising other requirements such as latency or availability for new writes. The fallback is that users may be temporary confused as they observe their own actions as not being immediately reflected. For example, they might change their skin or language preference but still get a one or two page views afterward with the old settings. Or they might have published an edit and briefly not yet see it appear in their contribution history.

Operational requirements

These are the expectations a site administrator must meet for chronology protection:

If the application is run from multiple data centers, then you must designate one of them as the "primary DC". The primary DC is where the primary database is located, from which replication propagates to replica databases in that same DC and any other DCs.
Web requests that use the POST verb, or carry a UseDC=master cookie, must be routed to the primary DC only.

An exception is requests carrying the Promise-Non-Write-API-Action: true header, which use the POST verb for large read queries, but don't actually require the primary DC.

If you have legacy extensions deployed that perform queries on the primary database during GET requests, then you will have to identify a way to route any of its relevant URLs to the primary DC as well, or to accept that their reads do not enjoy chronology protection, and that writes may be slower (due to cross-dc latency). See T91820 for Wikimedia Foundation's routing.

Access: internal

Definition at line 136 of file ChronologyProtector.php.

Constructor & Destructor Documentation

◆ __construct()

Wikimedia\Rdbms\ChronologyProtector::__construct	(	BagOStuff	$store,
		array	$client,
		?int	$clientPosIndex,
		string	$secret = `''`
	)

Parameters

BagOStuff	$store
array	$client	Map of (ip: <IP>, agent: <user-agent> [, clientId: <hash>] )
int \| null	$clientPosIndex	Write counter index of replication positions for this client
string	$secret	Secret string for HMAC hashing [optional]

Since: 1.27

Definition at line 193 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$store, and BagOStuff\makeGlobalKey().

Member Function Documentation

◆ applySessionReplicationPosition()

Wikimedia\Rdbms\ChronologyProtector::applySessionReplicationPosition ( ILoadBalancer $lb )

Apply client "session consistency" replication position to a new ILoadBalancer.

If the stash has a previous master position recorded, this will try to make sure that the next query to a replica DB of that master will see changes up to that position by delaying execution. The delay may timeout and allow stale data if no non-lagged replica DBs are available.

Access: internal: This method should only be called from LBFactory.

Parameters

ILoadBalancer $lb

Returns: void

Definition at line 261 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), Wikimedia\Rdbms\ILoadBalancer\getServerName(), Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions(), Wikimedia\Rdbms\ILoadBalancer\getWriterIndex(), and Wikimedia\Rdbms\ILoadBalancer\waitFor().

◆ getClientId()

Wikimedia\Rdbms\ChronologyProtector::getClientId ( )

Returns: string Client ID hash

Since: 1.32

Definition at line 228 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$clientId.

Referenced by Wikimedia\Rdbms\LBFactory\shutdown().

◆ getCurrentTime()

Wikimedia\Rdbms\ChronologyProtector::getCurrentTime ( )

protected

Access: internal: For testing only

Returns: float UNIX timestamp

Definition at line 552 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$wallClockOverride.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\stageSessionReplicationPosition().

◆ getStartupSessionPositions()

Wikimedia\Rdbms\ChronologyProtector::getStartupSessionPositions ( )

protected

Returns: array<string,DBMasterPos>

Definition at line 407 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$startupPositionsByMaster, and Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

Referenced by Wikimedia\Rdbms\ChronologyProtector\applySessionReplicationPosition().

◆ getStartupSessionTimestamps()

Wikimedia\Rdbms\ChronologyProtector::getStartupSessionTimestamps ( )

protected

Returns: array<string,float>

Definition at line 416 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$startupTimestampsByCluster, and Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

Referenced by Wikimedia\Rdbms\ChronologyProtector\getTouched().

◆ getTouched()

Wikimedia\Rdbms\ChronologyProtector::getTouched ( ILoadBalancer $lb )

Get the UNIX timestamp when the client last touched the DB, if they did so recently.

Access: internal: This method should only be called from LBFactory.

Parameters

ILoadBalancer $lb

Returns: float|false UNIX timestamp; false if not recent or on record

Since: 1.35

Definition at line 378 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), and Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ lazyStartup()

Wikimedia\Rdbms\ChronologyProtector::lazyStartup ( )

protected

Load the stored replication positions and touch timestamps for the client.

Returns: void

Definition at line 427 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\FLD_POSITIONS, Wikimedia\Rdbms\ChronologyProtector\FLD_TIMESTAMPS, Wikimedia\Rdbms\ChronologyProtector\FLD_WRITE_INDEX, Wikimedia\Rdbms\ChronologyProtector\getCurrentTime(), and Wikimedia\Rdbms\ChronologyProtector\POSITION_INDEX_WAIT_TIMEOUT.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions(), and Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ mergePositions()

Wikimedia\Rdbms\ChronologyProtector::mergePositions	(		$storedValue,
		array	$shutdownPositions,
		array	$shutdownTimestamps,
		?int &	$clientPosIndex = `null`
	)

protected

Merge the new replication positions with the currently stored ones (highest wins)

Parameters

	array<string,mixed>\|false	$storedValue Current replication position data
	array<string,DBMasterPos>	$shutdownPositions New replication positions
	array<string,float>	$shutdownTimestamps New DB post-commit shutdown timestamps
int \| null	&$clientPosIndex	New position write index

Returns: array<string,mixed> Combined replication position data

Definition at line 507 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\FLD_POSITIONS, Wikimedia\Rdbms\ChronologyProtector\FLD_TIMESTAMPS, and Wikimedia\Rdbms\ChronologyProtector\FLD_WRITE_INDEX.

Referenced by Wikimedia\Rdbms\ChronologyProtector\persistSessionReplicationPositions().

◆ persistSessionReplicationPositions()

Wikimedia\Rdbms\ChronologyProtector::persistSessionReplicationPositions ( & $clientPosIndex = null )

Persist any staged client "session consistency" replication positions.

Access: internal: This method should only be called from LBFactory.

Parameters

int | null &$clientPosIndex DB position key write counter; incremented on update

Returns: DBMasterPos[] Empty on success; map of (db name => unsaved position) on failure

Definition at line 321 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$shutdownPositionsByMaster, and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

Referenced by Wikimedia\Rdbms\LBFactory\shutdownChronologyProtector().

◆ setEnabled()

Wikimedia\Rdbms\ChronologyProtector::setEnabled ( $enabled )

Parameters

bool $enabled Whether reading/writing session replication positions is enabled

Since: 1.27

Definition at line 236 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$enabled.

◆ setLogger()

Wikimedia\Rdbms\ChronologyProtector::setLogger ( LoggerInterface $logger )

Definition at line 220 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$logger.

◆ setMockTime()

Wikimedia\Rdbms\ChronologyProtector::setMockTime ( & $time )

Access: internal: For testing only

Parameters

float | null &$time Mock UNIX timestamp

Definition at line 568 of file ChronologyProtector.php.

◆ setWaitEnabled()

Wikimedia\Rdbms\ChronologyProtector::setWaitEnabled ( $enabled )

Parameters

bool $enabled Whether session replication position wait barriers are enable

Since: 1.27

Definition at line 244 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$enabled.

◆ stageSessionReplicationPosition()

Wikimedia\Rdbms\ChronologyProtector::stageSessionReplicationPosition ( ILoadBalancer $lb )

Update client "session consistency" replication position for an end-of-life ILoadBalancer.

This remarks the replication position of the primary DB if this request made writes to it using the provided ILoadBalancer instance.

Access: internal: This method should only be called from LBFactory.

Parameters

ILoadBalancer $lb

Returns: void

Definition at line 289 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), Wikimedia\Rdbms\ChronologyProtector\getCurrentTime(), Wikimedia\Rdbms\ILoadBalancer\getReplicaResumePos(), Wikimedia\Rdbms\ILoadBalancer\getServerName(), Wikimedia\Rdbms\ILoadBalancer\getWriterIndex(), Wikimedia\Rdbms\ILoadBalancer\hasOrMadeRecentMasterChanges(), and Wikimedia\Rdbms\ILoadBalancer\hasStreamingReplicaServers().

Referenced by Wikimedia\Rdbms\LBFactory\shutdownChronologyProtector().

Member Data Documentation

◆ $clientId

string Wikimedia\Rdbms\ChronologyProtector::$clientId

protected

Hash of client parameters.

Definition at line 145 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getClientId().

◆ $clientLogInfo

string [] Wikimedia\Rdbms\ChronologyProtector::$clientLogInfo

protected

Map of client information fields for logging.

Definition at line 147 of file ChronologyProtector.php.

◆ $enabled

bool Wikimedia\Rdbms\ChronologyProtector::$enabled = true

protected

Whether reading/writing session consistency replication positions is enabled.

Definition at line 152 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\setEnabled(), and Wikimedia\Rdbms\ChronologyProtector\setWaitEnabled().

◆ $key

string Wikimedia\Rdbms\ChronologyProtector::$key

protected

Storage key name.

Definition at line 143 of file ChronologyProtector.php.

◆ $logger

LoggerInterface Wikimedia\Rdbms\ChronologyProtector::$logger

protected

Definition at line 140 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\setLogger().

◆ $positionWaitsEnabled

bool Wikimedia\Rdbms\ChronologyProtector::$positionWaitsEnabled = true

protected

Whether waiting on DB servers to reach replication positions is enabled.

Definition at line 154 of file ChronologyProtector.php.

◆ $shutdownPositionsByMaster

array<string,DBMasterPos> Wikimedia\Rdbms\ChronologyProtector::$shutdownPositionsByMaster = []

protected

Map of (DB master name => position)

Definition at line 161 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\persistSessionReplicationPositions().

◆ $shutdownTimestampsByCluster

array<string,float> Wikimedia\Rdbms\ChronologyProtector::$shutdownTimestampsByCluster = []

protected

Map of (DB cluster name => UNIX timestamp)

Definition at line 165 of file ChronologyProtector.php.

◆ $startupPositionsByMaster

array<string,DBMasterPos> Wikimedia\Rdbms\ChronologyProtector::$startupPositionsByMaster = []

protected

Map of (DB master name => position)

Definition at line 159 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions().

◆ $startupTimestamp

float null Wikimedia\Rdbms\ChronologyProtector::$startupTimestamp

protected

UNIX timestamp when the client data was loaded.

Definition at line 156 of file ChronologyProtector.php.

◆ $startupTimestampsByCluster

array<string,float> Wikimedia\Rdbms\ChronologyProtector::$startupTimestampsByCluster = []

protected

Map of (DB cluster name => UNIX timestamp)

Definition at line 163 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ $store

BagOStuff Wikimedia\Rdbms\ChronologyProtector::$store

protected

Definition at line 138 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\__construct().

◆ $waitForPosIndex

int null Wikimedia\Rdbms\ChronologyProtector::$waitForPosIndex

protected

Expected minimum index of the last write to the position store.

Definition at line 149 of file ChronologyProtector.php.

◆ $wallClockOverride

float null Wikimedia\Rdbms\ChronologyProtector::$wallClockOverride

private

Definition at line 168 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getCurrentTime().

◆ FLD_POSITIONS

const Wikimedia\Rdbms\ChronologyProtector::FLD_POSITIONS = 'positions'

private

Definition at line 182 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ FLD_TIMESTAMPS

const Wikimedia\Rdbms\ChronologyProtector::FLD_TIMESTAMPS = 'timestamps'

private

Definition at line 183 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ FLD_WRITE_INDEX

const Wikimedia\Rdbms\ChronologyProtector::FLD_WRITE_INDEX = 'writeIndex'

private

Definition at line 184 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ LOCK_TIMEOUT

const Wikimedia\Rdbms\ChronologyProtector::LOCK_TIMEOUT = 3

private

Lock timeout to use for key updates.

Definition at line 178 of file ChronologyProtector.php.

◆ LOCK_TTL

const Wikimedia\Rdbms\ChronologyProtector::LOCK_TTL = 6

private

Lock expiry to use for key updates.

Definition at line 180 of file ChronologyProtector.php.

◆ POSITION_COOKIE_TTL

const Wikimedia\Rdbms\ChronologyProtector::POSITION_COOKIE_TTL = 10

Seconds to store position write index cookies (safely less than POSITION_STORE_TTL)

Definition at line 171 of file ChronologyProtector.php.

◆ POSITION_INDEX_WAIT_TIMEOUT

const Wikimedia\Rdbms\ChronologyProtector::POSITION_INDEX_WAIT_TIMEOUT = 5

private

Max seconds to wait for positions write indexes to appear (e.g.

replicate) in storage

Definition at line 175 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

◆ POSITION_STORE_TTL

const Wikimedia\Rdbms\ChronologyProtector::POSITION_STORE_TTL = 60

private

Seconds to store replication positions.

Definition at line 173 of file ChronologyProtector.php.

The documentation for this class was generated from the following file:

includes/libs/rdbms/ChronologyProtector.php

Public Member Functions

Public Attributes

Protected Member Functions

Protected Attributes

Private Attributes

Detailed Description

In a nut shell

Purpose

How

Storage requirements

Operational requirements

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ applySessionReplicationPosition()

◆ getClientId()

◆ getCurrentTime()

◆ getStartupSessionPositions()

◆ getStartupSessionTimestamps()

◆ getTouched()

◆ lazyStartup()

◆ mergePositions()

◆ persistSessionReplicationPositions()

◆ setEnabled()

◆ setLogger()

◆ setMockTime()

◆ setWaitEnabled()

◆ stageSessionReplicationPosition()

Member Data Documentation

◆ $clientId

◆ $clientLogInfo

◆ $enabled

◆ $key

◆ $logger

◆ $positionWaitsEnabled

◆ $shutdownPositionsByMaster

◆ $shutdownTimestampsByCluster

◆ $startupPositionsByMaster

◆ $startupTimestamp

◆ $startupTimestampsByCluster

◆ $store

◆ $waitForPosIndex

◆ $wallClockOverride

◆ FLD_POSITIONS

◆ FLD_TIMESTAMPS

◆ FLD_WRITE_INDEX

◆ LOCK_TIMEOUT

◆ LOCK_TTL

◆ POSITION_COOKIE_TTL

◆ POSITION_INDEX_WAIT_TIMEOUT

◆ POSITION_STORE_TTL