MediaWiki  master
FileBackendMultiWrite Class Reference

Proxy backend that mirrors writes to several internal backends. More...

Inheritance diagram for FileBackendMultiWrite:
Collaboration diagram for FileBackendMultiWrite:

Public Member Functions

 __construct (array $config)
 Construct a proxy backend that consists of several internal backends. More...
 
 accessibilityCheck (array $paths)
 Check that a set of file paths are usable across all internal backends. More...
 
 clearCache (array $paths=null)
 
 concatenate (array $params)
 
 consistencyCheck (array $paths)
 Check that a set of files are consistent across all internal backends. More...
 
 directoryExists (array $params)
 
 fileExists (array $params)
 
 getDirectoryList (array $params)
 
 getFeatures ()
 
 getFileContentsMulti (array $params)
 
 getFileHttpUrl (array $params)
 
 getFileList (array $params)
 
 getFileProps (array $params)
 
 getFileSha1Base36 (array $params)
 
 getFileSize (array $params)
 
 getFileStat (array $params)
 
 getFileTimestamp (array $params)
 
 getFileXAttributes (array $params)
 
 getLocalCopyMulti (array $params)
 
 getLocalReferenceMulti (array $params)
 
 getScopedLocksForOps (array $ops, StatusValue $status)
 
 preloadCache (array $paths)
 
 preloadFileStat (array $params)
 
 resyncFiles (array $paths, $resyncMode=true)
 Check that a set of files are consistent across all internal backends and re-synchronize those files against the "multi master" if needed. More...
 
 streamFile (array $params)
 
- Public Member Functions inherited from FileBackend
 __construct (array $config)
 Create a new backend instance from configuration. More...
 
 clean (array $params)
 Delete a storage directory if it is empty. More...
 
 clearCache (array $paths=null)
 Invalidate any in-process file stat and property cache. More...
 
 concatenate (array $params)
 Concatenate a list of storage files into a single file system file. More...
 
 copy (array $params, array $opts=[])
 Performs a single copy operation. More...
 
 create (array $params, array $opts=[])
 Performs a single create operation. More...
 
 delete (array $params, array $opts=[])
 Performs a single delete operation. More...
 
 describe (array $params, array $opts=[])
 Performs a single describe operation. More...
 
 directoryExists (array $params)
 Check if a directory exists at a given storage path. More...
 
 doOperation (array $op, array $opts=[])
 Same as doOperations() except it takes a single operation. More...
 
 doOperations (array $ops, array $opts=[])
 This is the main entry point into the backend for write operations. More...
 
 doQuickOperation (array $op)
 Same as doQuickOperations() except it takes a single operation. More...
 
 doQuickOperations (array $ops, array $opts=[])
 Perform a set of independent file operations on some files. More...
 
 fileExists (array $params)
 Check if a file exists at a storage path in the backend. More...
 
 getContainerStoragePath ( $container)
 Get the storage path for the given container for this backend. More...
 
 getDirectoryList (array $params)
 Get an iterator to list all directories under a storage directory. More...
 
 getDomainId ()
 Get the domain identifier used for this backend (possibly empty). More...
 
 getFeatures ()
 Get the a bitfield of extra features supported by the backend medium. More...
 
 getFileContents (array $params)
 Get the contents of a file at a storage path in the backend. More...
 
 getFileContentsMulti (array $params)
 Like getFileContents() except it takes an array of storage paths and returns an order preserved map of storage paths to their content. More...
 
 getFileHttpUrl (array $params)
 Return an HTTP URL to a given file that requires no authentication to use. More...
 
 getFileList (array $params)
 Get an iterator to list all stored files under a storage directory. More...
 
 getFileProps (array $params)
 Get the properties of the content of the file at a storage path in the backend. More...
 
 getFileSha1Base36 (array $params)
 Get a SHA-1 hash of the content of the file at a storage path in the backend. More...
 
 getFileSize (array $params)
 Get the size (bytes) of a file at a storage path in the backend. More...
 
 getFileStat (array $params)
 Get quick information about a file at a storage path in the backend. More...
 
 getFileTimestamp (array $params)
 Get the last-modified timestamp of the file at a storage path. More...
 
 getFileXAttributes (array $params)
 Get metadata about a file at a storage path in the backend. More...
 
 getJournal ()
 Get the file journal object for this backend. More...
 
 getLocalCopy (array $params)
 Get a local copy on disk of the file at a storage path in the backend. More...
 
 getLocalCopyMulti (array $params)
 Like getLocalCopy() except it takes an array of storage paths and yields an order preserved-map of storage paths to temporary local file copies. More...
 
 getLocalReference (array $params)
 Returns a file system file, identical in content to the file at a storage path. More...
 
 getLocalReferenceMulti (array $params)
 Like getLocalReference() except it takes an array of storage paths and yields an order-preserved map of storage paths to temporary local file copies. More...
 
 getName ()
 Get the unique backend name. More...
 
 getReadOnlyReason ()
 Get an explanatory message if this backend is read-only. More...
 
 getRootStoragePath ()
 Get the root storage path of this backend. More...
 
 getScopedFileLocks (array $paths, $type, StatusValue $status, $timeout=0)
 Lock the files at the given storage paths in the backend. More...
 
 getScopedLocksForOps (array $ops, StatusValue $status)
 Get an array of scoped locks needed for a batch of file operations. More...
 
 getTopDirectoryList (array $params)
 Same as FileBackend::getDirectoryList() except only lists directories that are immediately under the given directory. More...
 
 getTopFileList (array $params)
 Same as FileBackend::getFileList() except only lists files that are immediately under the given directory. More...
 
 getWikiId ()
 Alias to getDomainId() More...
 
 hasFeatures ( $bitfield)
 Check if the backend medium supports a field of extra features. More...
 
 isReadOnly ()
 Check if this backend is read-only. More...
 
 lockFiles (array $paths, $type, $timeout=0)
 Lock the files at the given storage paths in the backend. More...
 
 move (array $params, array $opts=[])
 Performs a single move operation. More...
 
 preloadCache (array $paths)
 Preload persistent file stat cache and property cache into in-process cache. More...
 
 preloadFileStat (array $params)
 Preload file stat information (concurrently if possible) into in-process cache. More...
 
 prepare (array $params)
 Prepare a storage directory for usage. More...
 
 publish (array $params)
 Remove measures to block web access to a storage directory and the container it belongs to. More...
 
 quickCopy (array $params)
 Performs a single quick copy operation. More...
 
 quickCreate (array $params)
 Performs a single quick create operation. More...
 
 quickDelete (array $params)
 Performs a single quick delete operation. More...
 
 quickDescribe (array $params)
 Performs a single quick describe operation. More...
 
 quickMove (array $params)
 Performs a single quick move operation. More...
 
 quickStore (array $params)
 Performs a single quick store operation. More...
 
 secure (array $params)
 Take measures to block web access to a storage directory and the container it belongs to. More...
 
 setLogger (LoggerInterface $logger)
 
 store (array $params, array $opts=[])
 Performs a single store operation. More...
 
 streamFile (array $params)
 Stream the content of the file at a storage path in the backend. More...
 
 unlockFiles (array $paths, $type)
 Unlock the files at the given storage paths in the backend. More...
 

Protected Member Functions

 doClean (array $params)
 
 doDirectoryOp ( $method, array $params)
 
 doOperationsInternal (array $ops, array $opts)
 
 doPrepare (array $params)
 
 doPublish (array $params)
 
 doQuickOperationsInternal (array $ops)
 
 doSecure (array $params)
 
 fileStoragePathsForOps (array $ops)
 Get a list of file storage paths to read or write for a list of operations. More...
 
 getReadIndexFromParams (array $params)
 
 hasVolatileSources (array $ops)
 
 substOpBatchPaths (array $ops, FileBackendStore $backend)
 Substitute the backend name in storage path parameters for a set of operations with that of a given internal backend. More...
 
 substOpPaths (array $ops, FileBackendStore $backend)
 Same as substOpBatchPaths() but for a single operation. More...
 
 substPaths ( $paths, FileBackendStore $backend)
 Substitute the backend of storage paths with an internal backend's name. More...
 
 unsubstPaths ( $paths, FileBackendStore $backend)
 Substitute the backend of internal storage paths with the proxy backend's name. More...
 
- Protected Member Functions inherited from FileBackend
 doClean (array $params)
 
 doOperationsInternal (array $ops, array $opts)
 
 doPrepare (array $params)
 
 doPublish (array $params)
 
 doQuickOperationsInternal (array $ops)
 
 doSecure (array $params)
 
 newStatus (... $args)
 Yields the result of the status wrapper callback on either: More...
 
 resetOutputBuffer ()
 
 resolveFSFileObjects (array $ops)
 Convert FSFile 'src' paths to string paths (with an 'srcRef' field set to the FSFile) More...
 
 scopedProfileSection ( $section)
 
 wrapStatus (StatusValue $sv)
 

Protected Attributes

bool $asyncWrites = false
 
string bool $autoResync = false
 
FileBackendStore [] $backends = []
 Prioritized list of FileBackendStore objects. More...
 
int $masterIndex = -1
 Index of master backend. More...
 
int $readIndex = -1
 Index of read affinity backend. More...
 
int $syncChecks = 0
 Bitfield. More...
 
- Protected Attributes inherited from FileBackend
int $concurrency
 How many operations can be done in parallel. More...
 
string $domainId
 Unique domain name. More...
 
FileJournal $fileJournal
 
LockManager $lockManager
 
LoggerInterface $logger
 
string $name
 Unique backend name. More...
 
callable $obResetFunc
 
string $parallelize
 When to do operations in parallel. More...
 
callable null $profiler
 
string $readOnly
 Read-only explanation message. More...
 
callable $statusWrapper
 
callable $streamMimeFunc
 
TempFSFileFactory $tmpFileFactory
 

Additional Inherited Members

- Static Public Member Functions inherited from FileBackend
static extensionFromPath ( $path, $case='lowercase')
 Get the final extension from a storage or FS path. More...
 
static isPathTraversalFree ( $path)
 Check if a relative path has no directory traversals. More...
 
static isStoragePath ( $path)
 Check if a given path is a "mwstore://" path. More...
 
static makeContentDisposition ( $type, $filename='')
 Build a Content-Disposition header value per RFC 6266. More...
 
static normalizeStoragePath ( $storagePath)
 Normalize a storage path by cleaning up directory separators. More...
 
static parentStoragePath ( $storagePath)
 Get the parent storage directory of a storage path. More...
 
static splitStoragePath ( $storagePath)
 Split a storage path into a backend name, a container name, and a relative file path. More...
 
- Public Attributes inherited from FileBackend
const ATTR_HEADERS = 1
 Bitfield flags for supported features. More...
 
const ATTR_METADATA = 2
 
const ATTR_UNICODE_PATHS = 4
 
- Static Protected Member Functions inherited from FileBackend
static normalizeContainerPath ( $path)
 Validate and normalize a relative storage path. More...
 

Detailed Description

Proxy backend that mirrors writes to several internal backends.

This class defines a multi-write backend. Multiple backends can be registered to this proxy backend and it will act as a single backend. Use this when all access to those backends is through this proxy backend. At least one of the backends must be declared the "master" backend.

Only use this class when transitioning from one storage system to another.

Read operations are only done on the 'master' backend for consistency. Write operations are performed on all backends, starting with the master. This makes a best-effort to have transactional semantics, but since requests may sometimes fail, the use of "autoResync" or background scripts to fix inconsistencies is important.

Since
1.19

Definition at line 45 of file FileBackendMultiWrite.php.

Constructor & Destructor Documentation

◆ __construct()

FileBackendMultiWrite::__construct ( array  $config)

Construct a proxy backend that consists of several internal backends.

Locking, journaling, and read-only checks are handled by the proxy backend.

Additional $config params include:

  • backends : Array of backend config and multi-backend settings. Each value is the config used in the constructor of a FileBackendStore class, but with these additional settings:
    • class : The name of the backend class
    • isMultiMaster : This must be set for one backend.
    • readAffinity : Use this for reads without 'latest' set.
  • syncChecks : Integer bitfield of internal backend sync checks to perform. Possible bits include the FileBackendMultiWrite::CHECK_* constants. There are constants for SIZE, TIME, and SHA1. The checks are done before allowing any file operations.
  • autoResync : Automatically resync the clone backends to the master backend when pre-operation sync checks fail. This should only be used if the master backend is stable and not missing any files. Use "conservative" to limit resyncing to copying newer master backend files over older (or non-existing) clone backend files. Cases that cannot be handled will result in operation abortion.
  • replication : Set to 'async' to defer file operations on the non-master backends. This will apply such updates post-send for web requests. Note that any checks from "syncChecks" are still synchronous.
Parameters
array$config
Exceptions
LogicException

Definition at line 97 of file FileBackendMultiWrite.php.

References FileBackend\$domainId, FileBackend\$fileJournal, $masterIndex, and FileBackend\$name.

Member Function Documentation

◆ accessibilityCheck()

FileBackendMultiWrite::accessibilityCheck ( array  $paths)

Check that a set of file paths are usable across all internal backends.

Parameters
array$pathsList of storage paths
Returns
StatusValue

Definition at line 321 of file FileBackendMultiWrite.php.

References $path, FileBackend\newStatus(), and substPaths().

Referenced by doOperationsInternal().

◆ clearCache()

FileBackendMultiWrite::clearCache ( array  $paths = null)

Definition at line 763 of file FileBackendMultiWrite.php.

References substPaths().

Referenced by doOperationsInternal().

◆ concatenate()

FileBackendMultiWrite::concatenate ( array  $params)

◆ consistencyCheck()

FileBackendMultiWrite::consistencyCheck ( array  $paths)

Check that a set of files are consistent across all internal backends.

This method should only be called if the files are locked or the backend is in read-only mode

Parameters
array$pathsList of storage paths
Returns
StatusValue

Definition at line 233 of file FileBackendMultiWrite.php.

References $masterIndex, $path, FileBackend\newStatus(), substOpPaths(), substPaths(), and true.

Referenced by doOperationsInternal().

◆ directoryExists()

FileBackendMultiWrite::directoryExists ( array  $params)

Definition at line 741 of file FileBackendMultiWrite.php.

References $masterIndex, and substOpPaths().

◆ doClean()

FileBackendMultiWrite::doClean ( array  $params)
protected

Definition at line 589 of file FileBackendMultiWrite.php.

References doDirectoryOp().

◆ doDirectoryOp()

FileBackendMultiWrite::doDirectoryOp (   $method,
array  $params 
)
protected
Parameters
string$methodOne of (doPrepare,doSecure,doPublish,doClean)
array$paramsMethod arguments
Returns
StatusValue

Definition at line 598 of file FileBackendMultiWrite.php.

References $masterIndex, DeferredUpdates\addCallableUpdate(), FileBackend\newStatus(), and substOpPaths().

Referenced by doClean(), doPrepare(), doPublish(), and doSecure().

◆ doOperationsInternal()

◆ doPrepare()

FileBackendMultiWrite::doPrepare ( array  $params)
protected

Definition at line 577 of file FileBackendMultiWrite.php.

References doDirectoryOp().

◆ doPublish()

FileBackendMultiWrite::doPublish ( array  $params)
protected

Definition at line 585 of file FileBackendMultiWrite.php.

References doDirectoryOp().

◆ doQuickOperationsInternal()

FileBackendMultiWrite::doQuickOperationsInternal ( array  $ops)
protected

◆ doSecure()

FileBackendMultiWrite::doSecure ( array  $params)
protected

Definition at line 581 of file FileBackendMultiWrite.php.

References doDirectoryOp().

◆ fileExists()

FileBackendMultiWrite::fileExists ( array  $params)

Definition at line 636 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

Referenced by fileStoragePathsForOps().

◆ fileStoragePathsForOps()

FileBackendMultiWrite::fileStoragePathsForOps ( array  $ops)
protected

Get a list of file storage paths to read or write for a list of operations.

Parameters
array$opsSame format as doOperations()
Returns
array List of storage paths to files (does not include directories)

Definition at line 441 of file FileBackendMultiWrite.php.

References fileExists().

Referenced by doOperationsInternal().

◆ getDirectoryList()

FileBackendMultiWrite::getDirectoryList ( array  $params)

Definition at line 747 of file FileBackendMultiWrite.php.

References $masterIndex, and substOpPaths().

◆ getFeatures()

FileBackendMultiWrite::getFeatures ( )

Definition at line 759 of file FileBackendMultiWrite.php.

References $masterIndex.

◆ getFileContentsMulti()

FileBackendMultiWrite::getFileContentsMulti ( array  $params)

◆ getFileHttpUrl()

FileBackendMultiWrite::getFileHttpUrl ( array  $params)

Definition at line 734 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ getFileList()

FileBackendMultiWrite::getFileList ( array  $params)

Definition at line 753 of file FileBackendMultiWrite.php.

References $masterIndex, and substOpPaths().

◆ getFileProps()

FileBackendMultiWrite::getFileProps ( array  $params)

Definition at line 692 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ getFileSha1Base36()

FileBackendMultiWrite::getFileSha1Base36 ( array  $params)

Definition at line 685 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ getFileSize()

FileBackendMultiWrite::getFileSize ( array  $params)

Definition at line 650 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ getFileStat()

FileBackendMultiWrite::getFileStat ( array  $params)

Definition at line 657 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ getFileTimestamp()

FileBackendMultiWrite::getFileTimestamp ( array  $params)

Definition at line 643 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ getFileXAttributes()

FileBackendMultiWrite::getFileXAttributes ( array  $params)

Definition at line 664 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ getLocalCopyMulti()

FileBackendMultiWrite::getLocalCopyMulti ( array  $params)

◆ getLocalReferenceMulti()

FileBackendMultiWrite::getLocalReferenceMulti ( array  $params)

◆ getReadIndexFromParams()

FileBackendMultiWrite::getReadIndexFromParams ( array  $params)
protected
Parameters
array$params
Returns
int The master or read affinity backend index, based on $params['latest']

Definition at line 807 of file FileBackendMultiWrite.php.

References $readIndex.

Referenced by concatenate(), fileExists(), getFileContentsMulti(), getFileHttpUrl(), getFileProps(), getFileSha1Base36(), getFileSize(), getFileStat(), getFileTimestamp(), getFileXAttributes(), getLocalCopyMulti(), getLocalReferenceMulti(), preloadFileStat(), and streamFile().

◆ getScopedLocksForOps()

FileBackendMultiWrite::getScopedLocksForOps ( array  $ops,
StatusValue  $status 
)

◆ hasVolatileSources()

FileBackendMultiWrite::hasVolatileSources ( array  $ops)
protected
Parameters
array[]$ops File operations for FileBackend::doOperations()
Returns
bool Whether there are file path sources with outside lifetime/ownership

Definition at line 534 of file FileBackendMultiWrite.php.

Referenced by doOperationsInternal(), and doQuickOperationsInternal().

◆ preloadCache()

FileBackendMultiWrite::preloadCache ( array  $paths)

Definition at line 770 of file FileBackendMultiWrite.php.

References $readIndex, and substPaths().

◆ preloadFileStat()

FileBackendMultiWrite::preloadFileStat ( array  $params)

Definition at line 775 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ resyncFiles()

FileBackendMultiWrite::resyncFiles ( array  $paths,
  $resyncMode = true 
)

Check that a set of files are consistent across all internal backends and re-synchronize those files against the "multi master" if needed.

This method should only be called if the files are locked

Parameters
array$pathsList of storage paths
string | bool$resyncModeFalse, True, or "conservative"; see __construct()
Returns
StatusValue

Definition at line 349 of file FileBackendMultiWrite.php.

References $masterIndex, $path, FormatJson\encode(), FileBackend\newStatus(), substOpPaths(), and true.

Referenced by doOperationsInternal().

◆ streamFile()

FileBackendMultiWrite::streamFile ( array  $params)

Definition at line 699 of file FileBackendMultiWrite.php.

References getReadIndexFromParams(), and substOpPaths().

◆ substOpBatchPaths()

FileBackendMultiWrite::substOpBatchPaths ( array  $ops,
FileBackendStore  $backend 
)
protected

Substitute the backend name in storage path parameters for a set of operations with that of a given internal backend.

Parameters
array$opsList of file operation arrays
FileBackendStore$backend
Returns
array

Definition at line 472 of file FileBackendMultiWrite.php.

References substPaths().

Referenced by doOperationsInternal(), doQuickOperationsInternal(), getScopedLocksForOps(), and substOpPaths().

◆ substOpPaths()

◆ substPaths()

FileBackendMultiWrite::substPaths (   $paths,
FileBackendStore  $backend 
)
protected

Substitute the backend of storage paths with an internal backend's name.

Parameters
array | string$pathsList of paths or single string path
FileBackendStore$backend
Returns
string[]|string

Definition at line 507 of file FileBackendMultiWrite.php.

References StringUtils\escapeRegexReplacement().

Referenced by accessibilityCheck(), clearCache(), consistencyCheck(), preloadCache(), and substOpBatchPaths().

◆ unsubstPaths()

FileBackendMultiWrite::unsubstPaths (   $paths,
FileBackendStore  $backend 
)
protected

Substitute the backend of internal storage paths with the proxy backend's name.

Parameters
array | string$pathsList of paths or single string path
FileBackendStore$backendinternal storage backend
Returns
string[]|string

Definition at line 522 of file FileBackendMultiWrite.php.

References StringUtils\escapeRegexReplacement(), and FileBackend\getName().

Referenced by getFileContentsMulti(), getLocalCopyMulti(), getLocalReferenceMulti(), and getScopedLocksForOps().

Member Data Documentation

◆ $asyncWrites

bool FileBackendMultiWrite::$asyncWrites = false
protected

Definition at line 60 of file FileBackendMultiWrite.php.

◆ $autoResync

string bool FileBackendMultiWrite::$autoResync = false
protected

Definition at line 57 of file FileBackendMultiWrite.php.

◆ $backends

FileBackendStore [] FileBackendMultiWrite::$backends = []
protected

Prioritized list of FileBackendStore objects.

Definition at line 47 of file FileBackendMultiWrite.php.

◆ $masterIndex

int FileBackendMultiWrite::$masterIndex = -1
protected

◆ $readIndex

int FileBackendMultiWrite::$readIndex = -1
protected

Index of read affinity backend.

Definition at line 52 of file FileBackendMultiWrite.php.

Referenced by getReadIndexFromParams(), and preloadCache().

◆ $syncChecks

int FileBackendMultiWrite::$syncChecks = 0
protected

Bitfield.

Definition at line 55 of file FileBackendMultiWrite.php.


The documentation for this class was generated from the following file: