Maintenance script that moves blobs indexed by trackBlobs.php to a specified list of destination clusters, and recompresses them in the process. More...

Collaboration diagram for RecompressTracked:

[legend]

Public Member Functions
	__construct ( $options)

	critical ( $msg)

	debug ( $msg)

	execute ()
	Execute parent or child depending on the isChild option.

	executeChild ()
	Main entry point for worker processes.

	executeParent ()
	Execute the parent process.

	getTargetCluster ()
	Returns the name of the next target cluster.

	info ( $msg)

	moveTextRow ( $textId, $url)
	Atomic move operation.

Static Public Member Functions
static	getOptionsWithArgs ()

static	newFromCommandLine ( $args, $options)

Public Attributes
	$batchSize = 1000

	$childId = false

	$childPipes

	$childProcs

	$copyOnly = false

	$criticalLog

	$debugLog

	$destClusters

	$infoLog

	$isChild = false

	$noCount = false

	$numBatches = 0

	$numProcs = 1

	$orphanBatchSize = 1000

	$orphanBlobClass

	$pageBlobClass

	$prevChildId

	$reportingInterval = 10

ExternalStoreDB	$store

Private Member Functions
	checkTrackingTable ()
	Make sure the tracking table exists and isn't empty.

	dispatch (... $args)
	Dispatch a command to the next available child process.

	dispatchToChild ( $childId, $args)
	Dispatch a command to a specified child process.

	doAllOrphans ()
	Move all orphan text to the new clusters.

	doAllPages ()
	Move all tracked pages to the new clusters.

	doOrphanList ( $textIds)
	Move an orphan text_id to the new cluster.

	doPage ( $pageId)
	Move tracked text in a given page.

	finishIncompleteMoves ( $conds)
	Moves are done in two phases: bt_new_url and then bt_moved.

	getExtDB ( $cluster)
	Gets a DB master connection for the given external cluster name.

	killChildProcs ()
	Gracefully terminate the child processes.

	logToFile ( $msg, $file)

	report ( $label, $current, $end)
	Display a progress report.

	startChildProcs ()
	Start the worker processes.

	syncDBs ()
	Wait until the selected replica DB has caught up to the master.

Private Attributes
SqlBlobStore	$blobStore

Static Private Attributes
static	$cmdLineOptionMap

static	$optionsWithArgs

Detailed Description

Maintenance script that moves blobs indexed by trackBlobs.php to a specified list of destination clusters, and recompresses them in the process.

Definition at line 59 of file recompressTracked.php.

Constructor & Destructor Documentation

◆ __construct()

RecompressTracked::__construct ( $options )

Definition at line 112 of file recompressTracked.php.

References $GLOBALS.

Member Function Documentation

◆ checkTrackingTable()

RecompressTracked::checkTrackingTable ( )

private

Make sure the tracking table exists and isn't empty.

Returns: bool

Definition at line 204 of file recompressTracked.php.

References $dbr, critical(), DB_REPLICA, info(), and wfGetDB().

Referenced by executeParent().

◆ critical()

RecompressTracked::critical ( $msg )

Definition at line 146 of file recompressTracked.php.

References logToFile().

Referenced by checkTrackingTable(), dispatch(), doOrphanList(), doPage(), killChildProcs(), moveTextRow(), and startChildProcs().

◆ debug()

RecompressTracked::debug ( $msg )

Definition at line 132 of file recompressTracked.php.

References logToFile(), and wfDebug().

Referenced by doOrphanList(), doPage(), executeChild(), and finishIncompleteMoves().

◆ dispatch()

RecompressTracked::dispatch ( $args )

private

Dispatch a command to the next available child process.

This may block until a child process finishes its work and becomes available.

Parameters

array|string ...$args

Definition at line 287 of file recompressTracked.php.

References $args, $childId, $childPipes, $numProcs, critical(), and dispatchToChild().

Referenced by doAllOrphans(), and doAllPages().

◆ dispatchToChild()

RecompressTracked::dispatchToChild	(	$childId,
		$args
	)

private

Dispatch a command to a specified child process.

Parameters

int	$childId
array \| string	$args

Definition at line 314 of file recompressTracked.php.

References $args, and $childId.

Referenced by dispatch(), and killChildProcs().

◆ doAllOrphans()

RecompressTracked::doAllOrphans ( )

private

Move all orphan text to the new clusters.

Definition at line 392 of file recompressTracked.php.

References $args, $dbr, $res, DB_REPLICA, dispatch(), info(), report(), and wfGetDB().

Referenced by executeParent().

◆ doAllPages()

RecompressTracked::doAllPages ( )

private

Move all tracked pages to the new clusters.

Definition at line 323 of file recompressTracked.php.

References $dbr, $res, DB_REPLICA, dispatch(), info(), report(), and wfGetDB().

Referenced by executeParent().

◆ doOrphanList()

RecompressTracked::doOrphanList ( $textIds )

private

Move an orphan text_id to the new cluster.

Parameters

array $textIds

Definition at line 672 of file recompressTracked.php.

References $res, critical(), DB_REPLICA, debug(), finishIncompleteMoves(), syncDBs(), and wfGetDB().

Referenced by executeChild().

◆ doPage()

RecompressTracked::doPage ( $pageId )

private

Move tracked text in a given page.

Parameters

int $pageId

Definition at line 492 of file recompressTracked.php.

References $dbr, $res, $title, critical(), DB_REPLICA, debug(), finishIncompleteMoves(), syncDBs(), and wfGetDB().

Referenced by executeChild().

◆ execute()

RecompressTracked::execute ( )

Execute parent or child depending on the isChild option.

Definition at line 177 of file recompressTracked.php.

References executeChild(), and executeParent().

◆ executeChild()

RecompressTracked::executeChild ( )

Main entry point for worker processes.

Definition at line 461 of file recompressTracked.php.

References $args, $line, debug(), doOrphanList(), doPage(), and syncDBs().

Referenced by execute().

◆ executeParent()

RecompressTracked::executeParent ( )

Execute the parent process.

Definition at line 188 of file recompressTracked.php.

References checkTrackingTable(), doAllOrphans(), doAllPages(), killChildProcs(), startChildProcs(), and syncDBs().

Referenced by execute().

◆ finishIncompleteMoves()

RecompressTracked::finishIncompleteMoves ( $conds )

private

Moves are done in two phases: bt_new_url and then bt_moved.

bt_new_url indicates that the text has been copied to the new cluster.
bt_moved indicates that the text table has been updated.

This function completes any moves that only have done bt_new_url. This can happen when the script is interrupted, or when –copy-only is used.

Parameters

array $conds

Definition at line 609 of file recompressTracked.php.

References $dbr, $res, DB_REPLICA, debug(), moveTextRow(), and wfGetDB().

Referenced by doOrphanList(), and doPage().

◆ getExtDB()

RecompressTracked::getExtDB ( $cluster )

private

Gets a DB master connection for the given external cluster name.

Parameters

string $cluster

Returns: IMaintainableDatabase

Definition at line 660 of file recompressTracked.php.

References DB_MASTER.

◆ getOptionsWithArgs()

static RecompressTracked::getOptionsWithArgs ( )

static

Definition at line 97 of file recompressTracked.php.

References $optionsWithArgs.

◆ getTargetCluster()

RecompressTracked::getTargetCluster ( )

Returns the name of the next target cluster.

Returns: string

Definition at line 646 of file recompressTracked.php.

◆ info()

RecompressTracked::info ( $msg )

Definition at line 139 of file recompressTracked.php.

References logToFile().

Referenced by checkTrackingTable(), doAllOrphans(), doAllPages(), killChildProcs(), and report().

◆ killChildProcs()

RecompressTracked::killChildProcs ( )

private

Gracefully terminate the child processes.

Definition at line 268 of file recompressTracked.php.

References $numProcs, critical(), dispatchToChild(), and info().

Referenced by executeParent().

◆ logToFile()

RecompressTracked::logToFile	(	$msg,
		$file
	)

private

Definition at line 153 of file recompressTracked.php.

References $file, $header, and wfHostname().

Referenced by critical(), debug(), and info().

◆ moveTextRow()

RecompressTracked::moveTextRow	(	$textId,
		$url
	)

Atomic move operation.

Write the new URL to the text table and set the bt_moved flag.

This is done in a single transaction to provide restartable behavior without data loss.

The transaction is kept short to reduce locking.

Parameters

int	$textId
string	$url

Definition at line 574 of file recompressTracked.php.

References critical(), DB_MASTER, and wfGetDB().

Referenced by finishIncompleteMoves().

◆ newFromCommandLine()

static RecompressTracked::newFromCommandLine	(	$args,
		$options
	)

static

Definition at line 101 of file recompressTracked.php.

References $args.

◆ report()

RecompressTracked::report	(	$label,
		$current,
		$end
	)

private

Display a progress report.

Parameters

string	$label
int	$current
int	$end

Definition at line 380 of file recompressTracked.php.

References info().

Referenced by doAllOrphans(), and doAllPages().

◆ startChildProcs()

RecompressTracked::startChildProcs ( )

private

Start the worker processes.

These processes will listen on stdin for commands. This necessary because text recompression is slow: loading, compressing and writing are all slow.

Definition at line 227 of file recompressTracked.php.

References $numProcs, and critical().

Referenced by executeParent().

◆ syncDBs()

RecompressTracked::syncDBs ( )

private

Wait until the selected replica DB has caught up to the master.

This allows us to use the replica DB for things that were committed in a previous part of this batch process.

Definition at line 167 of file recompressTracked.php.

References $dbr, DB_MASTER, DB_REPLICA, and wfGetDB().

Referenced by doOrphanList(), doPage(), executeChild(), and executeParent().

Member Data Documentation

◆ $batchSize

RecompressTracked::$batchSize = 1000

Definition at line 61 of file recompressTracked.php.

◆ $blobStore

SqlBlobStore RecompressTracked::$blobStore

private

Definition at line 76 of file recompressTracked.php.

◆ $childId

RecompressTracked::$childId = false

Definition at line 70 of file recompressTracked.php.

Referenced by dispatch(), and dispatchToChild().

◆ $childPipes

RecompressTracked::$childPipes

Definition at line 67 of file recompressTracked.php.

Referenced by dispatch().

◆ $childProcs

RecompressTracked::$childProcs

Definition at line 67 of file recompressTracked.php.

◆ $cmdLineOptionMap

RecompressTracked::$cmdLineOptionMap

staticprivate

Initial value:

= [
        'no-count' => 'noCount',
        'procs' => 'numProcs',
        'copy-only' => 'copyOnly',
        'child' => 'isChild',
        'child-id' => 'childId',
        'debug-log' => 'debugLog',
        'info-log' => 'infoLog',
        'critical-log' => 'criticalLog',
    ]

Definition at line 86 of file recompressTracked.php.

◆ $copyOnly

RecompressTracked::$copyOnly = false

Definition at line 68 of file recompressTracked.php.

◆ $criticalLog

RecompressTracked::$criticalLog

Definition at line 72 of file recompressTracked.php.

◆ $debugLog

RecompressTracked::$debugLog

Definition at line 72 of file recompressTracked.php.

◆ $destClusters

RecompressTracked::$destClusters

Definition at line 60 of file recompressTracked.php.

◆ $infoLog

RecompressTracked::$infoLog

Definition at line 72 of file recompressTracked.php.

◆ $isChild

RecompressTracked::$isChild = false

Definition at line 69 of file recompressTracked.php.

◆ $noCount

RecompressTracked::$noCount = false

Definition at line 71 of file recompressTracked.php.

◆ $numBatches

RecompressTracked::$numBatches = 0

Definition at line 65 of file recompressTracked.php.

◆ $numProcs

RecompressTracked::$numProcs = 1

Definition at line 64 of file recompressTracked.php.

Referenced by dispatch(), killChildProcs(), and startChildProcs().

◆ $optionsWithArgs

RecompressTracked::$optionsWithArgs

staticprivate

Initial value:

= [
        'procs',
        'child-id',
        'debug-log',
        'info-log',
        'critical-log'
    ]

Definition at line 78 of file recompressTracked.php.

Referenced by getOptionsWithArgs().

◆ $orphanBatchSize

RecompressTracked::$orphanBatchSize = 1000

Definition at line 62 of file recompressTracked.php.

◆ $orphanBlobClass

RecompressTracked::$orphanBlobClass

Definition at line 66 of file recompressTracked.php.

◆ $pageBlobClass

RecompressTracked::$pageBlobClass

Definition at line 66 of file recompressTracked.php.

◆ $prevChildId

RecompressTracked::$prevChildId

Definition at line 67 of file recompressTracked.php.

◆ $reportingInterval

RecompressTracked::$reportingInterval = 10

Definition at line 63 of file recompressTracked.php.

◆ $store

ExternalStoreDB RecompressTracked::$store

Definition at line 74 of file recompressTracked.php.

The documentation for this class was generated from the following file:

maintenance/storage/recompressTracked.php

Public Member Functions

Static Public Member Functions

Public Attributes

Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ checkTrackingTable()

◆ critical()

◆ debug()

◆ dispatch()

◆ dispatchToChild()

◆ doAllOrphans()

◆ doAllPages()

◆ doOrphanList()

◆ doPage()

◆ execute()

◆ executeChild()

◆ executeParent()

◆ finishIncompleteMoves()

◆ getExtDB()

◆ getOptionsWithArgs()

◆ getTargetCluster()

◆ info()

◆ killChildProcs()

◆ logToFile()

◆ moveTextRow()

◆ newFromCommandLine()

◆ report()

◆ startChildProcs()

◆ syncDBs()

Member Data Documentation

◆ $batchSize

◆ $blobStore

◆ $childId

◆ $childPipes

◆ $childProcs

◆ $cmdLineOptionMap

◆ $copyOnly

◆ $criticalLog

◆ $debugLog

◆ $destClusters

◆ $infoLog

◆ $isChild

◆ $noCount

◆ $numBatches

◆ $numProcs

◆ $optionsWithArgs

◆ $orphanBatchSize

◆ $orphanBlobClass

◆ $pageBlobClass

◆ $prevChildId

◆ $reportingInterval

◆ $store