Provide a given client with protection against visible database lag. More...

Inheritance diagram for Wikimedia\Rdbms\ChronologyProtector:

Collaboration diagram for Wikimedia\Rdbms\ChronologyProtector:

[legend]

Public Member Functions
	__construct (BagOStuff $store, array $client, $posIndex, $secret='')

	applySessionReplicationPosition (ILoadBalancer $lb)
	Apply the "session consistency" DB replication position to a new ILoadBalancer. More...

	getClientId ()

	getTouched (ILoadBalancer $lb)
	Get the UNIX timestamp when the client last touched the DB, if they did so recently. More...

	setEnabled ( $enabled)

	setLogger (LoggerInterface $logger)

	setMockTime (&$time)

	setWaitEnabled ( $enabled)

	shutdown (&$cpIndex=null)
	Save any remarked "session consistency" DB replication positions to persistent storage. More...

	stageSessionReplicationPosition (ILoadBalancer $lb)
	Update the "session consistency" DB replication position for an end-of-life ILoadBalancer. More...

Public Attributes
const	POSITION_COOKIE_TTL = 10
	Seconds to store position write index cookies (safely less than POSITION_STORE_TTL) More...

Protected Member Functions
	getCurrentTime ()

	getStartupSessionPositions ()

	getStartupSessionTimestamps ()

	lazyStartup ()
	Load the stored DB replication positions and touch timestamps for the client. More...

	mergePositions ( $storedValue, array $shutdownPositions, array $shutdownTimestamps, &$cpIndex=null)
	Merge the new DB replication positions with the currently stored ones (highest wins) More...

Protected Attributes
string	$clientId
	Hash of client parameters. More...

string[]	$clientLogInfo
	Map of client information fields for logging. More...

bool	$enabled = true
	Whether reading/writing session consistency replication positions is enabled. More...

string	$key
	Storage key name. More...

LoggerInterface	$logger

bool	$positionWaitsEnabled = true
	Whether waiting on DB servers to reach replication positions is enabled. More...

array< string, DBMasterPos >	$shutdownPositionsByMaster = []
	Map of (DB master name => position) More...

array< string, float >	$shutdownTimestampsByCluster = []
	Map of (DB cluster name => UNIX timestamp) More...

array< string, DBMasterPos >	$startupPositionsByMaster = []
	Map of (DB master name => position) More...

float null	$startupTimestamp
	UNIX timestamp when the client data was loaded. More...

array< string, float >	$startupTimestampsByCluster = []
	Map of (DB cluster name => UNIX timestamp) More...

BagOStuff	$store

int null	$waitForPosIndex
	Expected minimum index of the last write to the position store. More...

Private Attributes
float null	$wallClockOverride

const	FLD_POSITIONS = 'positions'

const	FLD_TIMESTAMPS = 'timestamps'

const	FLD_WRITE_INDEX = 'writeIndex'

const	LOCK_TIMEOUT = 3
	Lock timeout to use for key updates. More...

const	LOCK_TTL = 6
	Lock expiry to use for key updates. More...

const	POSITION_INDEX_WAIT_TIMEOUT = 5
	Max seconds to wait for positions write indexes to appear (e.g. More...

const	POSITION_STORE_TTL = 60
	Seconds to store replication positions. More...

Detailed Description

Provide a given client with protection against visible database lag.

In a nut shell

This class tries to hide visible effects of database lag. It does this by temporarily remembering the database positions after a client makes a write, and on their next web request we will prefer non-lagged database replicas. When replica connections are establshed, we wait up to a few seconds for sufficient replication to have occurred, if they were not yet caught up to that same point.

This ensures a consistent ordering of events as seen by a client. Kind of like Hawking's Chronology Protection Agency.

Purpose

For performance and scalability reasons, almost all data is queried from replica databases. Only queries relating to writing data, are sent to a database master. When rendering a web page with content or activity feeds on it, the very latest information may thus not yet be there. That's okay in general, but if, for example, a client recently changed their preferences or submitted new data, we do our best to make sure their next web response does reflect at least their own recent changes.

How

To explain how it works, we will look at an example lifecycle for a client.

A client is browsing the site. Their web requests are generally read-only and display data from database replicas, which may be a few seconds out of date if a client elsewhere in the world recently modified that same data. If the application is run from multiple data centers, then these web requests may be served from the nearest secondary DC.

A client performs a POST request, perhaps to publish an edit or change their preferences. This request is routed to the primary DC (this is the responsibility of infrastructure outside the web app). There, the data is saved to the database master, after which the database host will asynchronously replicate this to its replicas in the same and any other DCs.

Toward the end of the response to this POST request, the application takes note of the database master's current "position", and save this under a "clientId" key in the ChronologyProtector store. The web response will also set two cookies that are similarly short-lived (about ten seconds): UseDC=master and cpPosIndex=<clientId>.

The ten seconds window is meant to account for the time needed for the database writes to have replicated across all active database replicas, including the cross-dc latency for those further away in any secondary DCs.

Future web requests from the client should fall in one of two categories:

Within the ten second window. Their UseDC cookie will make them return to the primary DC where we access the ChronologyProtector store and use the database "position" to decide which local database replica to use and on-demand wait a split second for replication to catch up if needed.
After the ten second window. They will be routed to the nearest and possibly different DC. Any local ChronologyProtector store existing there will not be interacted with. A random database replica may be used as the client's own writes are expected to have been applied here by now.

Storage requirements

The store used by ChronologyProtector, as configured via $wgChronologyProtectorStash, should meet the following requirements:

Low latencies. Nearly all web requests that involve a database connection will unconditionally query this store first. It is expected to respond within the order of one millisecond.
Best effort persistence, without active eviction pressure. Data stored here cannot be obtained elsewhere or recomputed. As such, under normal operating conditions, this store should not be full, and should not evict values before their intended expiry time ellapsed.
No replication, local consistency. Each DC may have a fully independent dc-local store associated with ChronologyProtector (no replication across DCs is needed). Local writes must be immediately reflected in subsequent local reads. No intra-dc read lag is allowed.
No redundancy, fast failure. Loss of data will likely be noticable and disruptive to clients, but the data is not considered essential. Under maintenance or unprecedented load, it is recommended to lose some data, instead of compromising other requirements such as latency or availability for new writes. The fallback is that users may be temporary confused as they observe their own actions as not being immediately reflected. For example, they might change their skin or language preference but still get a one or two page views afterward with the old settings. Or they might have published an edit and briefly not yet see it appear in their contribution history.

Operational requirements

These are the expectations a site administrator must meet for chronology protection:

If the application is run from multiple data centers, then you must designate one of them as the "primary DC". The primary DC is where the master database is located, from which replication propagates to replica databases in that same DC and any other DCs.
Web requests that use the POST verb, or carry a UseDC=master cookie, must be routed to the primary DC only.

An exception is requests carrying the Promise-Non-Write-API-Action: true header, which use the POST verb for large read queries, but don't actually require the primary DC.

If you have legacy extensions deployed that perform queries on the master database during GET requests, then you will have to identify a way to route any of its relevant URLs to the primary DC as well, or to accept that their reads do not enjoy chronology protection, and that writes may be slower (due to cross-dc latency). See T91820 for Wikimedia Foundation's routing.

Access: internal

Definition at line 135 of file ChronologyProtector.php.

Constructor & Destructor Documentation

◆ __construct()

Wikimedia\Rdbms\ChronologyProtector::__construct	(	BagOStuff	$store,
		array	$client,
			$posIndex,
			$secret = `''`
	)

Parameters

BagOStuff	$store
array	$client	Map of (ip: <IP>, agent: <user-agent> [, clientId: <hash>] )
int \| null	$posIndex	Write counter index
string	$secret	Secret string for HMAC hashing [optional]

Since: 1.27

Definition at line 192 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$store, and BagOStuff\makeGlobalKey().

Member Function Documentation

◆ applySessionReplicationPosition()

Wikimedia\Rdbms\ChronologyProtector::applySessionReplicationPosition ( ILoadBalancer $lb )

Apply the "session consistency" DB replication position to a new ILoadBalancer.

If the stash has a previous master position recorded, this will try to make sure that the next query to a replica DB of that master will see changes up to that position by delaying execution. The delay may timeout and allow stale data if no non-lagged replica DBs are available.

Access: internal: This method should only be called from LBFactory.

Parameters

ILoadBalancer $lb

Returns: void

Definition at line 255 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), Wikimedia\Rdbms\ILoadBalancer\getServerName(), Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions(), Wikimedia\Rdbms\ILoadBalancer\getWriterIndex(), and Wikimedia\Rdbms\ILoadBalancer\waitFor().

◆ getClientId()

Wikimedia\Rdbms\ChronologyProtector::getClientId ( )

Returns: string Client ID hash

Since: 1.32

Definition at line 222 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$clientId.

Referenced by Wikimedia\Rdbms\LBFactory\shutdown().

◆ getCurrentTime()

Wikimedia\Rdbms\ChronologyProtector::getCurrentTime ( )

protected

Access: internal: For testing only

Returns: float UNIX timestamp

Definition at line 546 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$wallClockOverride.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\stageSessionReplicationPosition().

◆ getStartupSessionPositions()

Wikimedia\Rdbms\ChronologyProtector::getStartupSessionPositions ( )

protected

Returns: array<string,DBMasterPos>

Definition at line 401 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$startupPositionsByMaster, and Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

Referenced by Wikimedia\Rdbms\ChronologyProtector\applySessionReplicationPosition().

◆ getStartupSessionTimestamps()

Wikimedia\Rdbms\ChronologyProtector::getStartupSessionTimestamps ( )

protected

Returns: array<string,float>

Definition at line 410 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$startupTimestampsByCluster, and Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

Referenced by Wikimedia\Rdbms\ChronologyProtector\getTouched().

◆ getTouched()

Wikimedia\Rdbms\ChronologyProtector::getTouched ( ILoadBalancer $lb )

Get the UNIX timestamp when the client last touched the DB, if they did so recently.

Access: internal: This method should only be called from LBFactory.

Parameters

ILoadBalancer $lb

Returns: float|false UNIX timestamp; false if not recent or on record

Since: 1.35

Definition at line 372 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), and Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ lazyStartup()

Wikimedia\Rdbms\ChronologyProtector::lazyStartup ( )

protected

Load the stored DB replication positions and touch timestamps for the client.

Returns: void

Definition at line 421 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\FLD_POSITIONS, Wikimedia\Rdbms\ChronologyProtector\FLD_TIMESTAMPS, Wikimedia\Rdbms\ChronologyProtector\FLD_WRITE_INDEX, Wikimedia\Rdbms\ChronologyProtector\getCurrentTime(), and Wikimedia\Rdbms\ChronologyProtector\POSITION_INDEX_WAIT_TIMEOUT.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions(), and Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ mergePositions()

Wikimedia\Rdbms\ChronologyProtector::mergePositions	(		$storedValue,
		array	$shutdownPositions,
		array	$shutdownTimestamps,
		&	$cpIndex = `null`
	)

protected

Merge the new DB replication positions with the currently stored ones (highest wins)

Parameters

	array<string,mixed>\|false	$storedValue Current DB replication position data
	array<string,DBMasterPos>	$shutdownPositions New DB replication positions
	array<string,float>	$shutdownTimestamps New DB post-commit shutdown timestamps
int \| null	&$cpIndex	New position write index

Returns: array<string,mixed> Combined DB replication position data

Definition at line 501 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\FLD_POSITIONS, Wikimedia\Rdbms\ChronologyProtector\FLD_TIMESTAMPS, and Wikimedia\Rdbms\ChronologyProtector\FLD_WRITE_INDEX.

Referenced by Wikimedia\Rdbms\ChronologyProtector\shutdown().

◆ setEnabled()

Wikimedia\Rdbms\ChronologyProtector::setEnabled ( $enabled )

Parameters

bool $enabled Whether reading/writing session replication positions is enabled

Since: 1.27

Definition at line 230 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$enabled.

◆ setLogger()

Wikimedia\Rdbms\ChronologyProtector::setLogger ( LoggerInterface $logger )

Definition at line 214 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$logger.

◆ setMockTime()

Wikimedia\Rdbms\ChronologyProtector::setMockTime ( & $time )

Access: internal: For testing only

Parameters

float | null &$time Mock UNIX timestamp

Definition at line 562 of file ChronologyProtector.php.

◆ setWaitEnabled()

Wikimedia\Rdbms\ChronologyProtector::setWaitEnabled ( $enabled )

Parameters

bool $enabled Whether session replication position wait barriers are enable

Since: 1.27

Definition at line 238 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$enabled.

◆ shutdown()

Wikimedia\Rdbms\ChronologyProtector::shutdown ( & $cpIndex = null )

Save any remarked "session consistency" DB replication positions to persistent storage.

Access: internal: This method should only be called from LBFactory.

Parameters

int | null &$cpIndex DB position key write counter; incremented on update

Returns: DBMasterPos[] Empty on success; map of (db name => unsaved position) on failure

Definition at line 315 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$shutdownPositionsByMaster, and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

Referenced by Wikimedia\Rdbms\LBFactory\shutdownChronologyProtector().

◆ stageSessionReplicationPosition()

Wikimedia\Rdbms\ChronologyProtector::stageSessionReplicationPosition ( ILoadBalancer $lb )

Update the "session consistency" DB replication position for an end-of-life ILoadBalancer.

This remarks the replication position of the master DB if this request made writes to it using the provided ILoadBalancer instance.

Access: internal: This method should only be called from LBFactory.

Parameters

ILoadBalancer $lb

Returns: void

Definition at line 283 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), Wikimedia\Rdbms\ChronologyProtector\getCurrentTime(), Wikimedia\Rdbms\ILoadBalancer\getReplicaResumePos(), Wikimedia\Rdbms\ILoadBalancer\getServerName(), Wikimedia\Rdbms\ILoadBalancer\getWriterIndex(), Wikimedia\Rdbms\ILoadBalancer\hasOrMadeRecentMasterChanges(), and Wikimedia\Rdbms\ILoadBalancer\hasStreamingReplicaServers().

Referenced by Wikimedia\Rdbms\LBFactory\shutdownChronologyProtector().

Member Data Documentation

◆ $clientId

string Wikimedia\Rdbms\ChronologyProtector::$clientId

protected

Hash of client parameters.

Definition at line 144 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getClientId().

◆ $clientLogInfo

string [] Wikimedia\Rdbms\ChronologyProtector::$clientLogInfo

protected

Map of client information fields for logging.

Definition at line 146 of file ChronologyProtector.php.

◆ $enabled

bool Wikimedia\Rdbms\ChronologyProtector::$enabled = true

protected

Whether reading/writing session consistency replication positions is enabled.

Definition at line 151 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\setEnabled(), and Wikimedia\Rdbms\ChronologyProtector\setWaitEnabled().

◆ $key

string Wikimedia\Rdbms\ChronologyProtector::$key

protected

Storage key name.

Definition at line 142 of file ChronologyProtector.php.

◆ $logger

LoggerInterface Wikimedia\Rdbms\ChronologyProtector::$logger

protected

Definition at line 139 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\setLogger().

◆ $positionWaitsEnabled

bool Wikimedia\Rdbms\ChronologyProtector::$positionWaitsEnabled = true

protected

Whether waiting on DB servers to reach replication positions is enabled.

Definition at line 153 of file ChronologyProtector.php.

◆ $shutdownPositionsByMaster

array<string,DBMasterPos> Wikimedia\Rdbms\ChronologyProtector::$shutdownPositionsByMaster = []

protected

Map of (DB master name => position)

Definition at line 160 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\shutdown().

◆ $shutdownTimestampsByCluster

array<string,float> Wikimedia\Rdbms\ChronologyProtector::$shutdownTimestampsByCluster = []

protected

Map of (DB cluster name => UNIX timestamp)

Definition at line 164 of file ChronologyProtector.php.

◆ $startupPositionsByMaster

array<string,DBMasterPos> Wikimedia\Rdbms\ChronologyProtector::$startupPositionsByMaster = []

protected

Map of (DB master name => position)

Definition at line 158 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions().

◆ $startupTimestamp

float null Wikimedia\Rdbms\ChronologyProtector::$startupTimestamp

protected

UNIX timestamp when the client data was loaded.

Definition at line 155 of file ChronologyProtector.php.

◆ $startupTimestampsByCluster

array<string,float> Wikimedia\Rdbms\ChronologyProtector::$startupTimestampsByCluster = []

protected

Map of (DB cluster name => UNIX timestamp)

Definition at line 162 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ $store

BagOStuff Wikimedia\Rdbms\ChronologyProtector::$store

protected

Definition at line 137 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\__construct().

◆ $waitForPosIndex

int null Wikimedia\Rdbms\ChronologyProtector::$waitForPosIndex

protected

Expected minimum index of the last write to the position store.

Definition at line 148 of file ChronologyProtector.php.

◆ $wallClockOverride

float null Wikimedia\Rdbms\ChronologyProtector::$wallClockOverride

private

Definition at line 167 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getCurrentTime().

◆ FLD_POSITIONS

const Wikimedia\Rdbms\ChronologyProtector::FLD_POSITIONS = 'positions'

private

Definition at line 181 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ FLD_TIMESTAMPS

const Wikimedia\Rdbms\ChronologyProtector::FLD_TIMESTAMPS = 'timestamps'

private

Definition at line 182 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ FLD_WRITE_INDEX

const Wikimedia\Rdbms\ChronologyProtector::FLD_WRITE_INDEX = 'writeIndex'

private

Definition at line 183 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ LOCK_TIMEOUT

const Wikimedia\Rdbms\ChronologyProtector::LOCK_TIMEOUT = 3

private

Lock timeout to use for key updates.

Definition at line 177 of file ChronologyProtector.php.

◆ LOCK_TTL

const Wikimedia\Rdbms\ChronologyProtector::LOCK_TTL = 6

private

Lock expiry to use for key updates.

Definition at line 179 of file ChronologyProtector.php.

◆ POSITION_COOKIE_TTL

const Wikimedia\Rdbms\ChronologyProtector::POSITION_COOKIE_TTL = 10

Seconds to store position write index cookies (safely less than POSITION_STORE_TTL)

Definition at line 170 of file ChronologyProtector.php.

◆ POSITION_INDEX_WAIT_TIMEOUT

const Wikimedia\Rdbms\ChronologyProtector::POSITION_INDEX_WAIT_TIMEOUT = 5

private

Max seconds to wait for positions write indexes to appear (e.g.

replicate) in storage

Definition at line 174 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

◆ POSITION_STORE_TTL

const Wikimedia\Rdbms\ChronologyProtector::POSITION_STORE_TTL = 60

private

Seconds to store replication positions.

Definition at line 172 of file ChronologyProtector.php.

The documentation for this class was generated from the following file:

includes/libs/rdbms/ChronologyProtector.php

Public Member Functions

Public Attributes

Protected Member Functions

Protected Attributes

Private Attributes

Detailed Description

In a nut shell

Purpose

How

Storage requirements

Operational requirements

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ applySessionReplicationPosition()

◆ getClientId()

◆ getCurrentTime()

◆ getStartupSessionPositions()

◆ getStartupSessionTimestamps()

◆ getTouched()

◆ lazyStartup()

◆ mergePositions()

◆ setEnabled()

◆ setLogger()

◆ setMockTime()

◆ setWaitEnabled()

◆ shutdown()

◆ stageSessionReplicationPosition()

Member Data Documentation

◆ $clientId

◆ $clientLogInfo

◆ $enabled

◆ $key

◆ $logger

◆ $positionWaitsEnabled

◆ $shutdownPositionsByMaster

◆ $shutdownTimestampsByCluster

◆ $startupPositionsByMaster

◆ $startupTimestamp

◆ $startupTimestampsByCluster

◆ $store

◆ $waitForPosIndex

◆ $wallClockOverride

◆ FLD_POSITIONS

◆ FLD_TIMESTAMPS

◆ FLD_WRITE_INDEX

◆ LOCK_TIMEOUT

◆ LOCK_TTL

◆ POSITION_COOKIE_TTL

◆ POSITION_INDEX_WAIT_TIMEOUT

◆ POSITION_STORE_TTL