MediaWiki REL1_37
Public Member Functions | Public Attributes | Protected Member Functions | Protected Attributes | Private Attributes | List of all members
Wikimedia\Rdbms\ChronologyProtector Class Reference

Provide a given client with protection against visible database lag. More...

Inheritance diagram for Wikimedia\Rdbms\ChronologyProtector:
Inheritance graph
[legend]
Collaboration diagram for Wikimedia\Rdbms\ChronologyProtector:
Collaboration graph
[legend]

Public Member Functions

 __construct (BagOStuff $store, array $client, ?int $clientPosIndex, string $secret='')
 
 applySessionReplicationPosition (ILoadBalancer $lb)
 Apply client "session consistency" replication position to a new ILoadBalancer.
 
 getClientId ()
 
 getTouched (ILoadBalancer $lb)
 Get the UNIX timestamp when the client last touched the DB, if they did so recently.
 
 persistSessionReplicationPositions (&$clientPosIndex=null)
 Persist any staged client "session consistency" replication positions.
 
 setEnabled ( $enabled)
 
 setLogger (LoggerInterface $logger)
 
 setMockTime (&$time)
 
 setWaitEnabled ( $enabled)
 
 stageSessionReplicationPosition (ILoadBalancer $lb)
 Update client "session consistency" replication position for an end-of-life ILoadBalancer.
 

Public Attributes

const POSITION_COOKIE_TTL = 10
 Seconds to store position write index cookies (safely less than POSITION_STORE_TTL)
 

Protected Member Functions

 getCurrentTime ()
 
 getStartupSessionPositions ()
 
 getStartupSessionTimestamps ()
 
 lazyStartup ()
 Load the stored replication positions and touch timestamps for the client.
 
 mergePositions ( $storedValue, array $shutdownPositions, array $shutdownTimestamps, ?int &$clientPosIndex=null)
 Merge the new replication positions with the currently stored ones (highest wins)
 

Protected Attributes

string $clientId
 Hash of client parameters.
 
string[] $clientLogInfo
 Map of client information fields for logging.
 
bool $enabled = true
 Whether reading/writing session consistency replication positions is enabled.
 
string $key
 Storage key name.
 
LoggerInterface $logger
 
bool $positionWaitsEnabled = true
 Whether waiting on DB servers to reach replication positions is enabled.
 
array< string, DBPrimaryPos > $shutdownPositionsByMaster = []
 Map of (DB primary name => position)
 
array< string, float > $shutdownTimestampsByCluster = []
 Map of (DB cluster name => UNIX timestamp)
 
array< string, DBPrimaryPos > $startupPositionsByMaster = []
 Map of (DB primary name => position)
 
float null $startupTimestamp
 UNIX timestamp when the client data was loaded.
 
array< string, float > $startupTimestampsByCluster = []
 Map of (DB cluster name => UNIX timestamp)
 
BagOStuff $store
 
int null $waitForPosIndex
 Expected minimum index of the last write to the position store.
 

Private Attributes

float null $wallClockOverride
 
const FLD_POSITIONS = 'positions'
 
const FLD_TIMESTAMPS = 'timestamps'
 
const FLD_WRITE_INDEX = 'writeIndex'
 
const LOCK_TIMEOUT = 3
 Lock timeout to use for key updates.
 
const LOCK_TTL = 6
 Lock expiry to use for key updates.
 
const POSITION_INDEX_WAIT_TIMEOUT = 5
 Max seconds to wait for positions write indexes to appear (e.g.
 
const POSITION_STORE_TTL = 60
 Seconds to store replication positions.
 

Detailed Description

Provide a given client with protection against visible database lag.

In a nut shell

This class tries to hide visible effects of database lag. It does this by temporarily remembering the database positions after a client makes a write, and on their next web request we will prefer non-lagged database replicas. When replica connections are establshed, we wait up to a few seconds for sufficient replication to have occurred, if they were not yet caught up to that same point.

This ensures a consistent ordering of events as seen by a client. Kind of like Hawking's Chronology Protection Agency.

Purpose

For performance and scalability reasons, almost all data is queried from replica databases. Only queries relating to writing data, are sent to a primary database. When rendering a web page with content or activity feeds on it, the very latest information may thus not yet be there. That's okay in general, but if, for example, a client recently changed their preferences or submitted new data, we do our best to make sure their next web response does reflect at least their own recent changes.

How

To explain how it works, we will look at an example lifecycle for a client.

A client is browsing the site. Their web requests are generally read-only and display data from database replicas, which may be a few seconds out of date if a client elsewhere in the world recently modified that same data. If the application is run from multiple data centers, then these web requests may be served from the nearest secondary DC.

A client performs a POST request, perhaps to publish an edit or change their preferences. This request is routed to the primary DC (this is the responsibility of infrastructure outside the web app). There, the data is saved to the primary database, after which the database host will asynchronously replicate this to its replicas in the same and any other DCs.

Toward the end of the response to this POST request, the application takes note of the primary database's current "position", and save this under a "clientId" key in the ChronologyProtector store. The web response will also set two cookies that are similarly short-lived (about ten seconds): UseDC=master and cpPosIndex=<posIndex>@<write time>#<clientId>.

The ten seconds window is meant to account for the time needed for the database writes to have replicated across all active database replicas, including the cross-dc latency for those further away in any secondary DCs. The "clientId" is placed in the cookie to handle the case where the client IP addresses frequently changes between web requests.

Future web requests from the client should fall in one of two categories:

  1. Within the ten second window. Their UseDC cookie will make them return to the primary DC where we access the ChronologyProtector store and use the database "position" to decide which local database replica to use and on-demand wait a split second for replication to catch up if needed.
  2. After the ten second window. They will be routed to the nearest and possibly different DC. Any local ChronologyProtector store existing there will not be interacted with. A random database replica may be used as the client's own writes are expected to have been applied here by now.

Storage requirements

The store used by ChronologyProtector, as configured via $wgChronologyProtectorStash, should meet the following requirements:

  • Low latencies. Nearly all web requests that involve a database connection will unconditionally query this store first. It is expected to respond within the order of one millisecond.
  • Best effort persistence, without active eviction pressure. Data stored here cannot be obtained elsewhere or recomputed. As such, under normal operating conditions, this store should not be full, and should not evict values before their intended expiry time ellapsed.
  • No replication, local consistency. Each DC may have a fully independent dc-local store associated with ChronologyProtector (no replication across DCs is needed). Local writes must be immediately reflected in subsequent local reads. No intra-dc read lag is allowed.
  • No redundancy, fast failure. Loss of data will likely be noticable and disruptive to clients, but the data is not considered essential. Under maintenance or unprecedented load, it is recommended to lose some data, instead of compromising other requirements such as latency or availability for new writes. The fallback is that users may be temporary confused as they observe their own actions as not being immediately reflected. For example, they might change their skin or language preference but still get a one or two page views afterward with the old settings. Or they might have published an edit and briefly not yet see it appear in their contribution history.

Operational requirements

These are the expectations a site administrator must meet for chronology protection:

  • If the application is run from multiple data centers, then you must designate one of them as the "primary DC". The primary DC is where the primary database is located, from which replication propagates to replica databases in that same DC and any other DCs.

  • Web requests that use the POST verb, or carry a UseDC=master cookie, must be routed to the primary DC only.

    An exception is requests carrying the Promise-Non-Write-API-Action: true header, which use the POST verb for large read queries, but don't actually require the primary DC.

    If you have legacy extensions deployed that perform queries on the primary database during GET requests, then you will have to identify a way to route any of its relevant URLs to the primary DC as well, or to accept that their reads do not enjoy chronology protection, and that writes may be slower (due to cross-dc latency). See T91820 for Wikimedia Foundation's routing.

Access: internal

Definition at line 136 of file ChronologyProtector.php.

Constructor & Destructor Documentation

◆ __construct()

Wikimedia\Rdbms\ChronologyProtector::__construct ( BagOStuff  $store,
array  $client,
?int  $clientPosIndex,
string  $secret = '' 
)
Parameters
BagOStuff$store
array$clientMap of (ip: <IP>, agent: <user-agent> [, clientId: <hash>] )
int | null$clientPosIndexWrite counter index of replication positions for this client
string$secretSecret string for HMAC hashing [optional]
Since
1.27

Definition at line 193 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$store, and BagOStuff\makeGlobalKey().

Member Function Documentation

◆ applySessionReplicationPosition()

Wikimedia\Rdbms\ChronologyProtector::applySessionReplicationPosition ( ILoadBalancer  $lb)

Apply client "session consistency" replication position to a new ILoadBalancer.

If the stash has a previous primary position recorded, this will try to make sure that the next query to a replica DB of that primary DB will see changes up to that position by delaying execution. The delay may timeout and allow stale data if no non-lagged replica DBs are available.

Access: internal
This method should only be called from LBFactory.
Parameters
ILoadBalancer$lb
Returns
void

Definition at line 261 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), Wikimedia\Rdbms\ILoadBalancer\getServerName(), Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions(), Wikimedia\Rdbms\ILoadBalancer\getWriterIndex(), and Wikimedia\Rdbms\ILoadBalancer\waitFor().

◆ getClientId()

Wikimedia\Rdbms\ChronologyProtector::getClientId ( )
Returns
string Client ID hash
Since
1.32

Definition at line 228 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$clientId.

Referenced by Wikimedia\Rdbms\LBFactory\shutdown().

◆ getCurrentTime()

Wikimedia\Rdbms\ChronologyProtector::getCurrentTime ( )
protected
Access: internal
For testing only
Returns
float UNIX timestamp

Definition at line 552 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$wallClockOverride.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\stageSessionReplicationPosition().

◆ getStartupSessionPositions()

Wikimedia\Rdbms\ChronologyProtector::getStartupSessionPositions ( )
protected
Returns
array<string,DBPrimaryPos>

Definition at line 407 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$startupPositionsByMaster, and Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

Referenced by Wikimedia\Rdbms\ChronologyProtector\applySessionReplicationPosition().

◆ getStartupSessionTimestamps()

Wikimedia\Rdbms\ChronologyProtector::getStartupSessionTimestamps ( )
protected
Returns
array<string,float>

Definition at line 416 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$startupTimestampsByCluster, and Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

Referenced by Wikimedia\Rdbms\ChronologyProtector\getTouched().

◆ getTouched()

Wikimedia\Rdbms\ChronologyProtector::getTouched ( ILoadBalancer  $lb)

Get the UNIX timestamp when the client last touched the DB, if they did so recently.

Access: internal
This method should only be called from LBFactory.
Parameters
ILoadBalancer$lb
Returns
float|false UNIX timestamp; false if not recent or on record
Since
1.35

Definition at line 378 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), and Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ lazyStartup()

Wikimedia\Rdbms\ChronologyProtector::lazyStartup ( )
protected

Load the stored replication positions and touch timestamps for the client.

Returns
void

Definition at line 427 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\FLD_POSITIONS, Wikimedia\Rdbms\ChronologyProtector\FLD_TIMESTAMPS, Wikimedia\Rdbms\ChronologyProtector\FLD_WRITE_INDEX, Wikimedia\Rdbms\ChronologyProtector\getCurrentTime(), and Wikimedia\Rdbms\ChronologyProtector\POSITION_INDEX_WAIT_TIMEOUT.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions(), and Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ mergePositions()

Wikimedia\Rdbms\ChronologyProtector::mergePositions (   $storedValue,
array  $shutdownPositions,
array  $shutdownTimestamps,
?int &  $clientPosIndex = null 
)
protected

Merge the new replication positions with the currently stored ones (highest wins)

Parameters
array<string,mixed>|false$storedValue Current replication position data
array<string,DBPrimaryPos>$shutdownPositions New replication positions
array<string,float>$shutdownTimestamps New DB post-commit shutdown timestamps
int | null&$clientPosIndexNew position write index
Returns
array<string,mixed> Combined replication position data

Definition at line 507 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\FLD_POSITIONS, Wikimedia\Rdbms\ChronologyProtector\FLD_TIMESTAMPS, and Wikimedia\Rdbms\ChronologyProtector\FLD_WRITE_INDEX.

Referenced by Wikimedia\Rdbms\ChronologyProtector\persistSessionReplicationPositions().

◆ persistSessionReplicationPositions()

Wikimedia\Rdbms\ChronologyProtector::persistSessionReplicationPositions ( &  $clientPosIndex = null)

Persist any staged client "session consistency" replication positions.

Access: internal
This method should only be called from LBFactory.
Parameters
int | null&$clientPosIndexDB position key write counter; incremented on update
Returns
DBPrimaryPos[] Empty on success; map of (db name => unsaved position) on failure

Definition at line 321 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$shutdownPositionsByMaster, and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

Referenced by Wikimedia\Rdbms\LBFactory\shutdownChronologyProtector().

◆ setEnabled()

Wikimedia\Rdbms\ChronologyProtector::setEnabled (   $enabled)
Parameters
bool$enabledWhether reading/writing session replication positions is enabled
Since
1.27

Definition at line 236 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$enabled.

◆ setLogger()

Wikimedia\Rdbms\ChronologyProtector::setLogger ( LoggerInterface  $logger)

Definition at line 220 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$logger.

Referenced by Wikimedia\Rdbms\LBFactory\getChronologyProtector().

◆ setMockTime()

Wikimedia\Rdbms\ChronologyProtector::setMockTime ( &  $time)
Access: internal
For testing only
Parameters
float | null&$timeMock UNIX timestamp

Definition at line 568 of file ChronologyProtector.php.

◆ setWaitEnabled()

Wikimedia\Rdbms\ChronologyProtector::setWaitEnabled (   $enabled)
Parameters
bool$enabledWhether session replication position wait barriers are enable
Since
1.27

Definition at line 244 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ChronologyProtector\$enabled.

◆ stageSessionReplicationPosition()

Wikimedia\Rdbms\ChronologyProtector::stageSessionReplicationPosition ( ILoadBalancer  $lb)

Update client "session consistency" replication position for an end-of-life ILoadBalancer.

This remarks the replication position of the primary DB if this request made writes to it using the provided ILoadBalancer instance.

Access: internal
This method should only be called from LBFactory.
Parameters
ILoadBalancer$lb
Returns
void

Definition at line 289 of file ChronologyProtector.php.

References Wikimedia\Rdbms\ILoadBalancer\getClusterName(), Wikimedia\Rdbms\ChronologyProtector\getCurrentTime(), Wikimedia\Rdbms\ILoadBalancer\getReplicaResumePos(), Wikimedia\Rdbms\ILoadBalancer\getServerName(), Wikimedia\Rdbms\ILoadBalancer\getWriterIndex(), Wikimedia\Rdbms\ILoadBalancer\hasOrMadeRecentPrimaryChanges(), and Wikimedia\Rdbms\ILoadBalancer\hasStreamingReplicaServers().

Referenced by Wikimedia\Rdbms\LBFactory\shutdownChronologyProtector().

Member Data Documentation

◆ $clientId

string Wikimedia\Rdbms\ChronologyProtector::$clientId
protected

Hash of client parameters.

Definition at line 145 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getClientId().

◆ $clientLogInfo

string [] Wikimedia\Rdbms\ChronologyProtector::$clientLogInfo
protected

Map of client information fields for logging.

Definition at line 147 of file ChronologyProtector.php.

◆ $enabled

bool Wikimedia\Rdbms\ChronologyProtector::$enabled = true
protected

Whether reading/writing session consistency replication positions is enabled.

Definition at line 152 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\setEnabled(), and Wikimedia\Rdbms\ChronologyProtector\setWaitEnabled().

◆ $key

string Wikimedia\Rdbms\ChronologyProtector::$key
protected

Storage key name.

Definition at line 143 of file ChronologyProtector.php.

◆ $logger

LoggerInterface Wikimedia\Rdbms\ChronologyProtector::$logger
protected

Definition at line 140 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\setLogger().

◆ $positionWaitsEnabled

bool Wikimedia\Rdbms\ChronologyProtector::$positionWaitsEnabled = true
protected

Whether waiting on DB servers to reach replication positions is enabled.

Definition at line 154 of file ChronologyProtector.php.

◆ $shutdownPositionsByMaster

array<string,DBPrimaryPos> Wikimedia\Rdbms\ChronologyProtector::$shutdownPositionsByMaster = []
protected

Map of (DB primary name => position)

Definition at line 161 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\persistSessionReplicationPositions().

◆ $shutdownTimestampsByCluster

array<string,float> Wikimedia\Rdbms\ChronologyProtector::$shutdownTimestampsByCluster = []
protected

Map of (DB cluster name => UNIX timestamp)

Definition at line 165 of file ChronologyProtector.php.

◆ $startupPositionsByMaster

array<string,DBPrimaryPos> Wikimedia\Rdbms\ChronologyProtector::$startupPositionsByMaster = []
protected

Map of (DB primary name => position)

Definition at line 159 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionPositions().

◆ $startupTimestamp

float null Wikimedia\Rdbms\ChronologyProtector::$startupTimestamp
protected

UNIX timestamp when the client data was loaded.

Definition at line 156 of file ChronologyProtector.php.

◆ $startupTimestampsByCluster

array<string,float> Wikimedia\Rdbms\ChronologyProtector::$startupTimestampsByCluster = []
protected

Map of (DB cluster name => UNIX timestamp)

Definition at line 163 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getStartupSessionTimestamps().

◆ $store

BagOStuff Wikimedia\Rdbms\ChronologyProtector::$store
protected

Definition at line 138 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\__construct().

◆ $waitForPosIndex

int null Wikimedia\Rdbms\ChronologyProtector::$waitForPosIndex
protected

Expected minimum index of the last write to the position store.

Definition at line 149 of file ChronologyProtector.php.

◆ $wallClockOverride

float null Wikimedia\Rdbms\ChronologyProtector::$wallClockOverride
private

Definition at line 168 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\getCurrentTime().

◆ FLD_POSITIONS

const Wikimedia\Rdbms\ChronologyProtector::FLD_POSITIONS = 'positions'
private

Definition at line 182 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ FLD_TIMESTAMPS

const Wikimedia\Rdbms\ChronologyProtector::FLD_TIMESTAMPS = 'timestamps'
private

Definition at line 183 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ FLD_WRITE_INDEX

const Wikimedia\Rdbms\ChronologyProtector::FLD_WRITE_INDEX = 'writeIndex'
private

Definition at line 184 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup(), and Wikimedia\Rdbms\ChronologyProtector\mergePositions().

◆ LOCK_TIMEOUT

const Wikimedia\Rdbms\ChronologyProtector::LOCK_TIMEOUT = 3
private

Lock timeout to use for key updates.

Definition at line 178 of file ChronologyProtector.php.

◆ LOCK_TTL

const Wikimedia\Rdbms\ChronologyProtector::LOCK_TTL = 6
private

Lock expiry to use for key updates.

Definition at line 180 of file ChronologyProtector.php.

◆ POSITION_COOKIE_TTL

const Wikimedia\Rdbms\ChronologyProtector::POSITION_COOKIE_TTL = 10

Seconds to store position write index cookies (safely less than POSITION_STORE_TTL)

Definition at line 171 of file ChronologyProtector.php.

◆ POSITION_INDEX_WAIT_TIMEOUT

const Wikimedia\Rdbms\ChronologyProtector::POSITION_INDEX_WAIT_TIMEOUT = 5
private

Max seconds to wait for positions write indexes to appear (e.g.

replicate) in storage

Definition at line 175 of file ChronologyProtector.php.

Referenced by Wikimedia\Rdbms\ChronologyProtector\lazyStartup().

◆ POSITION_STORE_TTL

const Wikimedia\Rdbms\ChronologyProtector::POSITION_STORE_TTL = 60
private

Seconds to store replication positions.

Definition at line 173 of file ChronologyProtector.php.

The documentation for this class was generated from the following file: