Maintenance script that moves blobs indexed by trackBlobs.php to a specified list of destination clusters, and recompresses them in the process. More...

Public Member Functions
	__construct ( $options)

	checkTrackingTable ()
	Make sure the tracking table exists and isn't empty.

	critical ( $msg)

	debug ( $msg)

	dispatch ()
	Dispatch a command to the next available slave.

	dispatchToSlave ( $slaveId, $args)
	Dispatch a command to a specified slave.

	doAllOrphans ()
	Move all orphan text to the new clusters.

	doAllPages ()
	Move all tracked pages to the new clusters.

	doOrphanList ( $textIds)
	Move an orphan text_id to the new cluster.

	doPage ( $pageId)
	Move tracked text in a given page.

	execute ()
	Execute parent or child depending on the isChild option.

	executeChild ()
	Main entry point for worker processes.

	executeParent ()
	Execute the parent process.

	finishIncompleteMoves ( $conds)
	Moves are done in two phases: bt_new_url and then bt_moved.

	getExtDB ( $cluster)
	Gets a DB master connection for the given external cluster name.

	getTargetCluster ()
	Returns the name of the next target cluster.

	info ( $msg)

	killSlaveProcs ()
	Gracefully terminate the child processes.

	logToFile ( $msg, $file)

	moveTextRow ( $textId, $url)
	Atomic move operation.

	report ( $label, $current, $end)
	Display a progress report.

	startSlaveProcs ()
	Start the worker processes.

	syncDBs ()
	Wait until the selected slave has caught up to the master.

Static Public Member Functions
static	getOptionsWithArgs ()

static	newFromCommandLine ( $args, $options)

Public Attributes
	$batchSize = 1000

	$copyOnly = false

	$criticalLog

	$debugLog

	$destClusters

	$infoLog

	$isChild = false

	$noCount = false

	$numBatches = 0

	$numProcs = 1

	$orphanBatchSize = 1000

	$orphanBlobClass

	$pageBlobClass

	$prevSlaveId

	$reportingInterval = 10

	$slaveId = false

	$slavePipes

	$slaveProcs

	$store

	$useDiff

Static Private Attributes
static	$cmdLineOptionMap

static	$optionsWithArgs

Detailed Description

Maintenance script that moves blobs indexed by trackBlobs.php to a specified list of destination clusters, and recompresses them in the process.

Definition at line 55 of file recompressTracked.php.

Constructor & Destructor Documentation

◆ __construct()

RecompressTracked::__construct ( $options )

Definition at line 105 of file recompressTracked.php.

References $GLOBALS, $name, $options, $value, as, and store.

Member Function Documentation

◆ checkTrackingTable()

RecompressTracked::checkTrackingTable ( )

Make sure the tracking table exists and isn't empty.

Returns: bool

Definition at line 192 of file recompressTracked.php.

References $dbr, critical(), DB_SLAVE, info(), and wfGetDB().

Referenced by executeParent().

◆ critical()

RecompressTracked::critical ( $msg )

Definition at line 134 of file recompressTracked.php.

References logToFile().

Referenced by checkTrackingTable(), dispatch(), doOrphanList(), doPage(), killSlaveProcs(), moveTextRow(), and startSlaveProcs().

◆ debug()

RecompressTracked::debug ( $msg )

Definition at line 120 of file recompressTracked.php.

References logToFile(), and wfDebug().

Referenced by doOrphanList(), doPage(), executeChild(), and finishIncompleteMoves().

◆ dispatch()

RecompressTracked::dispatch ( )

Dispatch a command to the next available slave.

This may block until a slave finishes its work and becomes available.

Definition at line 272 of file recompressTracked.php.

References $args, $i, $numProcs, $slaveId, $slavePipes, critical(), and dispatchToSlave().

Referenced by doAllPages().

◆ dispatchToSlave()

RecompressTracked::dispatchToSlave	(	$slaveId,
		$args
	)

Dispatch a command to a specified slave.

Parameters

int	$slaveId
array \| string	$args

Definition at line 298 of file recompressTracked.php.

References $args, $slaveId, and array().

Referenced by dispatch(), and killSlaveProcs().

◆ doAllOrphans()

RecompressTracked::doAllOrphans ( )

Move all orphan text to the new clusters.

Definition at line 376 of file recompressTracked.php.

References $args, $dbr, $i, $res, as, DB_SLAVE, info(), report(), and wfGetDB().

Referenced by executeParent().

◆ doAllPages()

RecompressTracked::doAllPages ( )

Move all tracked pages to the new clusters.

Definition at line 307 of file recompressTracked.php.

References $dbr, $i, $res, as, DB_SLAVE, dispatch(), info(), is(), query, report(), so, that, the, and wfGetDB().

Referenced by executeParent().

◆ doOrphanList()

RecompressTracked::doOrphanList ( $textIds )

Move an orphan text_id to the new cluster.

Parameters

array $textIds

Definition at line 653 of file recompressTracked.php.

References $res, as, critical(), DB_SLAVE, debug(), finishIncompleteMoves(), Revision\getRevisionText(), syncDBs(), wfGetDB(), and wfWaitForSlaves().

Referenced by executeChild().

◆ doPage()

RecompressTracked::doPage ( $pageId )

Move tracked text in a given page.

Parameters

int $pageId

Definition at line 476 of file recompressTracked.php.

References $dbr, $res, $title, as, critical(), DB_SLAVE, debug(), finishIncompleteMoves(), Revision\getRevisionText(), Title\newFromID(), syncDBs(), wfGetDB(), and wfWaitForSlaves().

Referenced by executeChild().

◆ execute()

RecompressTracked::execute ( )

Execute parent or child depending on the isChild option.

Definition at line 165 of file recompressTracked.php.

References executeChild(), and executeParent().

◆ executeChild()

RecompressTracked::executeChild ( )

Main entry point for worker processes.

Definition at line 445 of file recompressTracked.php.

References $args, $line, debug(), doOrphanList(), doPage(), syncDBs(), and wfWaitForSlaves().

Referenced by execute().

◆ executeParent()

RecompressTracked::executeParent ( )

Execute the parent process.

Definition at line 176 of file recompressTracked.php.

References checkTrackingTable(), doAllOrphans(), doAllPages(), killSlaveProcs(), startSlaveProcs(), and syncDBs().

Referenced by execute().

◆ finishIncompleteMoves()

RecompressTracked::finishIncompleteMoves ( $conds )

Moves are done in two phases: bt_new_url and then bt_moved.

bt_new_url indicates that the text has been copied to the new cluster.
bt_moved indicates that the text table has been updated.

This function completes any moves that only have done bt_new_url. This can happen when the script is interrupted, or when –copy-only is used.

Parameters

array $conds

Definition at line 592 of file recompressTracked.php.

References $dbr, $res, as, DB_SLAVE, debug(), moveTextRow(), wfGetDB(), and wfWaitForSlaves().

Referenced by doOrphanList(), and doPage().

◆ getExtDB()

RecompressTracked::getExtDB ( $cluster )

Gets a DB master connection for the given external cluster name.

Parameters

string $cluster

Returns: DatabaseBase

Definition at line 642 of file recompressTracked.php.

References DB_MASTER, and wfGetLBFactory().

◆ getOptionsWithArgs()

static RecompressTracked::getOptionsWithArgs ( )

static

Definition at line 90 of file recompressTracked.php.

References $optionsWithArgs.

◆ getTargetCluster()

RecompressTracked::getTargetCluster ( )

Returns the name of the next target cluster.

Returns: string

Definition at line 628 of file recompressTracked.php.

◆ info()

RecompressTracked::info ( $msg )

Definition at line 127 of file recompressTracked.php.

References logToFile().

Referenced by checkTrackingTable(), doAllOrphans(), doAllPages(), killSlaveProcs(), and report().

◆ killSlaveProcs()

RecompressTracked::killSlaveProcs ( )

Gracefully terminate the child processes.

Definition at line 254 of file recompressTracked.php.

References $i, $numProcs, $status, critical(), dispatchToSlave(), and info().

Referenced by executeParent().

◆ logToFile()

RecompressTracked::logToFile	(	$msg,
		$file
	)

Definition at line 141 of file recompressTracked.php.

References wfHostname(), and wfWikiID().

Referenced by critical(), debug(), and info().

◆ moveTextRow()

RecompressTracked::moveTextRow	(	$textId,
		$url
	)

Atomic move operation.

Write the new URL to the text table and set the bt_moved flag.

This is done in a single transaction to provide restartable behavior without data loss.

The transaction is kept short to reduce locking.

Parameters

int	$textId
string	$url

Definition at line 557 of file recompressTracked.php.

References critical(), DB_MASTER, and wfGetDB().

Referenced by finishIncompleteMoves().

◆ newFromCommandLine()

static RecompressTracked::newFromCommandLine	(	$args,
		$options
	)

static

Definition at line 94 of file recompressTracked.php.

References $args, $options, and as.

◆ report()

RecompressTracked::report	(	$label,
		$current,
		$end
	)

Display a progress report.

Parameters

string	$label
int	$current
int	$end

Definition at line 364 of file recompressTracked.php.

References info(), and wfWaitForSlaves().

Referenced by doAllOrphans(), and doAllPages().

◆ startSlaveProcs()

RecompressTracked::startSlaveProcs ( )

Start the worker processes.

These processes will listen on stdin for commands. This necessary because text recompression is slow: loading, compressing and writing are all slow.

Definition at line 215 of file recompressTracked.php.

References $i, $numProcs, as, critical(), wfEscapeShellArg(), and wfWikiID().

Referenced by executeParent().

◆ syncDBs()

RecompressTracked::syncDBs ( )

Wait until the selected slave has caught up to the master.

This allows us to use the slave for things that were committed in a previous part of this batch process.

Definition at line 155 of file recompressTracked.php.

References $dbr, DB_MASTER, DB_SLAVE, and wfGetDB().

Referenced by doOrphanList(), doPage(), executeChild(), and executeParent().

Member Data Documentation

◆ $batchSize

RecompressTracked::$batchSize = 1000

Definition at line 57 of file recompressTracked.php.

◆ $cmdLineOptionMap

RecompressTracked::$cmdLineOptionMap

staticprivate

Initial value:

= [
        'no-count' => 'noCount',
        'procs' => 'numProcs',
        'copy-only' => 'copyOnly',
        'child' => 'isChild',
        'slave-id' => 'slaveId',
        'debug-log' => 'debugLog',
        'info-log' => 'infoLog',
        'critical-log' => 'criticalLog',
    ]

Definition at line 79 of file recompressTracked.php.

◆ $copyOnly

RecompressTracked::$copyOnly = false

Definition at line 64 of file recompressTracked.php.

◆ $criticalLog

RecompressTracked::$criticalLog

Definition at line 68 of file recompressTracked.php.

◆ $debugLog

RecompressTracked::$debugLog

Definition at line 68 of file recompressTracked.php.

◆ $destClusters

RecompressTracked::$destClusters

Definition at line 56 of file recompressTracked.php.

◆ $infoLog

RecompressTracked::$infoLog

Definition at line 68 of file recompressTracked.php.

◆ $isChild

RecompressTracked::$isChild = false

Definition at line 65 of file recompressTracked.php.

◆ $noCount

RecompressTracked::$noCount = false

Definition at line 67 of file recompressTracked.php.

◆ $numBatches

RecompressTracked::$numBatches = 0

Definition at line 61 of file recompressTracked.php.

◆ $numProcs

RecompressTracked::$numProcs = 1

Definition at line 60 of file recompressTracked.php.

Referenced by dispatch(), killSlaveProcs(), and startSlaveProcs().

◆ $optionsWithArgs

RecompressTracked::$optionsWithArgs

staticprivate

Initial value:

= [
        'procs',
        'slave-id',
        'debug-log',
        'info-log',
        'critical-log'
    ]

Definition at line 71 of file recompressTracked.php.

Referenced by getOptionsWithArgs().

◆ $orphanBatchSize

RecompressTracked::$orphanBatchSize = 1000

Definition at line 58 of file recompressTracked.php.

◆ $orphanBlobClass

RecompressTracked::$orphanBlobClass

Definition at line 62 of file recompressTracked.php.

◆ $pageBlobClass

RecompressTracked::$pageBlobClass

Definition at line 62 of file recompressTracked.php.

◆ $prevSlaveId

RecompressTracked::$prevSlaveId

Definition at line 63 of file recompressTracked.php.

◆ $reportingInterval

RecompressTracked::$reportingInterval = 10

Definition at line 59 of file recompressTracked.php.

◆ $slaveId

RecompressTracked::$slaveId = false

Definition at line 66 of file recompressTracked.php.

Referenced by dispatch(), and dispatchToSlave().

◆ $slavePipes

RecompressTracked::$slavePipes

Definition at line 63 of file recompressTracked.php.

Referenced by dispatch().

◆ $slaveProcs

RecompressTracked::$slaveProcs

Definition at line 63 of file recompressTracked.php.

◆ $store

RecompressTracked::$store

Definition at line 69 of file recompressTracked.php.

◆ $useDiff

RecompressTracked::$useDiff

Definition at line 62 of file recompressTracked.php.

The documentation for this class was generated from the following file:

maintenance/storage/recompressTracked.php

Public Member Functions

Static Public Member Functions

Public Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ __construct()

Member Function Documentation

◆ checkTrackingTable()

◆ critical()

◆ debug()

◆ dispatch()

◆ dispatchToSlave()

◆ doAllOrphans()

◆ doAllPages()

◆ doOrphanList()

◆ doPage()

◆ execute()

◆ executeChild()

◆ executeParent()

◆ finishIncompleteMoves()

◆ getExtDB()

◆ getOptionsWithArgs()

◆ getTargetCluster()

◆ info()

◆ killSlaveProcs()

◆ logToFile()

◆ moveTextRow()

◆ newFromCommandLine()

◆ report()

◆ startSlaveProcs()

◆ syncDBs()

Member Data Documentation

◆ $batchSize

◆ $cmdLineOptionMap

◆ $copyOnly

◆ $criticalLog

◆ $debugLog

◆ $destClusters

◆ $infoLog

◆ $isChild

◆ $noCount

◆ $numBatches

◆ $numProcs

◆ $optionsWithArgs

◆ $orphanBatchSize

◆ $orphanBlobClass

◆ $pageBlobClass

◆ $prevSlaveId

◆ $reportingInterval

◆ $slaveId

◆ $slavePipes

◆ $slaveProcs

◆ $store

◆ $useDiff