MediaWiki  master
RecompressTracked Class Reference

Maintenance script that moves blobs indexed by trackBlobs.php to a specified list of destination clusters, and recompresses them in the process. More...

Public Member Functions

 __construct ( $options)
 
 checkTrackingTable ()
 Make sure the tracking table exists and isn't empty. More...
 
 critical ( $msg)
 
 debug ( $msg)
 
 dispatch (... $args)
 Dispatch a command to the next available replica DB. More...
 
 dispatchToReplica ( $replicaId, $args)
 Dispatch a command to a specified replica DB. More...
 
 doAllOrphans ()
 Move all orphan text to the new clusters. More...
 
 doAllPages ()
 Move all tracked pages to the new clusters. More...
 
 doOrphanList ( $textIds)
 Move an orphan text_id to the new cluster. More...
 
 doPage ( $pageId)
 Move tracked text in a given page. More...
 
 execute ()
 Execute parent or child depending on the isChild option. More...
 
 executeChild ()
 Main entry point for worker processes. More...
 
 executeParent ()
 Execute the parent process. More...
 
 finishIncompleteMoves ( $conds)
 Moves are done in two phases: bt_new_url and then bt_moved. More...
 
 getExtDB ( $cluster)
 Gets a DB master connection for the given external cluster name. More...
 
 getTargetCluster ()
 Returns the name of the next target cluster. More...
 
 info ( $msg)
 
 killReplicaProcs ()
 Gracefully terminate the child processes. More...
 
 logToFile ( $msg, $file)
 
 moveTextRow ( $textId, $url)
 Atomic move operation. More...
 
 report ( $label, $current, $end)
 Display a progress report. More...
 
 startReplicaProcs ()
 Start the worker processes. More...
 
 syncDBs ()
 Wait until the selected replica DB has caught up to the master. More...
 

Static Public Member Functions

static static getOptionsWithArgs ()
 
static newFromCommandLine ( $args, $options)
 

Public Attributes

 $batchSize = 1000
 
 $copyOnly = false
 
 $criticalLog
 
 $debugLog
 
 $destClusters
 
 $infoLog
 
 $isChild = false
 
 $noCount = false
 
 $numBatches = 0
 
 $numProcs = 1
 
 $orphanBatchSize = 1000
 
 $orphanBlobClass
 
 $pageBlobClass
 
 $prevReplicaId
 
 $replicaId = false
 
 $replicaPipes
 
 $replicaProcs
 
 $reportingInterval = 10
 
 $store
 

Static Private Attributes

static $cmdLineOptionMap
 
static $optionsWithArgs
 

Detailed Description

Maintenance script that moves blobs indexed by trackBlobs.php to a specified list of destination clusters, and recompresses them in the process.

Definition at line 58 of file recompressTracked.php.

Constructor & Destructor Documentation

◆ __construct()

RecompressTracked::__construct (   $options)

Definition at line 108 of file recompressTracked.php.

References $GLOBALS, $name, $options, $value, as, and class.

Member Function Documentation

◆ checkTrackingTable()

RecompressTracked::checkTrackingTable ( )

Make sure the tracking table exists and isn't empty.

Returns
bool

Definition at line 195 of file recompressTracked.php.

References $dbr, critical(), DB_REPLICA, info(), and wfGetDB().

Referenced by executeParent().

◆ critical()

RecompressTracked::critical (   $msg)

◆ debug()

RecompressTracked::debug (   $msg)

Definition at line 123 of file recompressTracked.php.

References logToFile(), and wfDebug().

Referenced by doOrphanList(), doPage(), executeChild(), and finishIncompleteMoves().

◆ dispatch()

RecompressTracked::dispatch (   $args)

Dispatch a command to the next available replica DB.

This may block until a replica DB finishes its work and becomes available.

Definition at line 275 of file recompressTracked.php.

References $args, $numProcs, $replicaId, $replicaPipes, critical(), and dispatchToReplica().

Referenced by doAllOrphans(), and doAllPages().

◆ dispatchToReplica()

RecompressTracked::dispatchToReplica (   $replicaId,
  $args 
)

Dispatch a command to a specified replica DB.

Parameters
int$replicaId
array | string$args

Definition at line 302 of file recompressTracked.php.

References $args, $replicaId, and array().

Referenced by dispatch(), and killReplicaProcs().

◆ doAllOrphans()

RecompressTracked::doAllOrphans ( )

Move all orphan text to the new clusters.

Definition at line 380 of file recompressTracked.php.

References $args, $batchSize, $dbr, $orphanBatchSize, $res, as, DB_REPLICA, dispatch(), info(), report(), and wfGetDB().

Referenced by executeParent().

◆ doAllPages()

RecompressTracked::doAllPages ( )

Move all tracked pages to the new clusters.

Definition at line 311 of file recompressTracked.php.

References $batchSize, $dbr, $res, as, DB_REPLICA, dispatch(), info(), is, query, report(), so, that, and wfGetDB().

Referenced by executeParent().

◆ doOrphanList()

RecompressTracked::doOrphanList (   $textIds)

Move an orphan text_id to the new cluster.

Parameters
array$textIds

Definition at line 660 of file recompressTracked.php.

References $res, as, critical(), DB_REPLICA, debug(), finishIncompleteMoves(), Revision\getRevisionText(), syncDBs(), and wfGetDB().

Referenced by executeChild().

◆ doPage()

RecompressTracked::doPage (   $pageId)

Move tracked text in a given page.

Parameters
int$pageId

Definition at line 480 of file recompressTracked.php.

References $batchSize, $dbr, $res, $title, as, critical(), DB_REPLICA, debug(), finishIncompleteMoves(), Revision\getRevisionText(), Title\newFromID(), syncDBs(), and wfGetDB().

Referenced by executeChild().

◆ execute()

RecompressTracked::execute ( )

Execute parent or child depending on the isChild option.

Definition at line 168 of file recompressTracked.php.

References executeChild(), and executeParent().

◆ executeChild()

RecompressTracked::executeChild ( )

Main entry point for worker processes.

Definition at line 449 of file recompressTracked.php.

References $args, $line, debug(), doOrphanList(), doPage(), and syncDBs().

Referenced by execute().

◆ executeParent()

RecompressTracked::executeParent ( )

Execute the parent process.

Definition at line 179 of file recompressTracked.php.

References checkTrackingTable(), doAllOrphans(), doAllPages(), killReplicaProcs(), startReplicaProcs(), and syncDBs().

Referenced by execute().

◆ finishIncompleteMoves()

RecompressTracked::finishIncompleteMoves (   $conds)

Moves are done in two phases: bt_new_url and then bt_moved.

  • bt_new_url indicates that the text has been copied to the new cluster.
  • bt_moved indicates that the text table has been updated.

This function completes any moves that only have done bt_new_url. This can happen when the script is interrupted, or when –copy-only is used.

Parameters
array$conds

Definition at line 597 of file recompressTracked.php.

References $dbr, $res, as, DB_REPLICA, debug(), moveTextRow(), and wfGetDB().

Referenced by doOrphanList(), and doPage().

◆ getExtDB()

RecompressTracked::getExtDB (   $cluster)

Gets a DB master connection for the given external cluster name.

Parameters
string$cluster
Returns
IDatabase

Definition at line 648 of file recompressTracked.php.

References DB_MASTER.

◆ getOptionsWithArgs()

static static RecompressTracked::getOptionsWithArgs ( )
static

Definition at line 93 of file recompressTracked.php.

References $optionsWithArgs.

◆ getTargetCluster()

RecompressTracked::getTargetCluster ( )

Returns the name of the next target cluster.

Returns
string

Definition at line 634 of file recompressTracked.php.

◆ info()

RecompressTracked::info (   $msg)

Definition at line 130 of file recompressTracked.php.

References logToFile().

Referenced by checkTrackingTable(), doAllOrphans(), doAllPages(), killReplicaProcs(), and report().

◆ killReplicaProcs()

RecompressTracked::killReplicaProcs ( )

Gracefully terminate the child processes.

Definition at line 257 of file recompressTracked.php.

References $numProcs, critical(), dispatchToReplica(), and info().

Referenced by executeParent().

◆ logToFile()

RecompressTracked::logToFile (   $msg,
  $file 
)

Definition at line 144 of file recompressTracked.php.

References $file, $header, wfHostname(), and wfWikiID().

Referenced by critical(), debug(), and info().

◆ moveTextRow()

RecompressTracked::moveTextRow (   $textId,
  $url 
)

Atomic move operation.

Write the new URL to the text table and set the bt_moved flag.

This is done in a single transaction to provide restartable behavior without data loss.

The transaction is kept short to reduce locking.

Parameters
int$textId
string$url

Definition at line 562 of file recompressTracked.php.

References critical(), DB_MASTER, and wfGetDB().

Referenced by finishIncompleteMoves().

◆ newFromCommandLine()

static RecompressTracked::newFromCommandLine (   $args,
  $options 
)
static

Definition at line 97 of file recompressTracked.php.

References $args, $options, and as.

◆ report()

RecompressTracked::report (   $label,
  $current,
  $end 
)

Display a progress report.

Parameters
string$label
int$current
int$end

Definition at line 368 of file recompressTracked.php.

References info().

Referenced by doAllOrphans(), and doAllPages().

◆ startReplicaProcs()

RecompressTracked::startReplicaProcs ( )

Start the worker processes.

These processes will listen on stdin for commands. This necessary because text recompression is slow: loading, compressing and writing are all slow.

Definition at line 218 of file recompressTracked.php.

References $numProcs, $optionsWithArgs, as, critical(), and wfWikiID().

Referenced by executeParent().

◆ syncDBs()

RecompressTracked::syncDBs ( )

Wait until the selected replica DB has caught up to the master.

This allows us to use the replica DB for things that were committed in a previous part of this batch process.

Definition at line 158 of file recompressTracked.php.

References $dbr, DB_MASTER, DB_REPLICA, and wfGetDB().

Referenced by doOrphanList(), doPage(), executeChild(), and executeParent().

Member Data Documentation

◆ $batchSize

RecompressTracked::$batchSize = 1000

Definition at line 60 of file recompressTracked.php.

Referenced by doAllOrphans(), doAllPages(), and doPage().

◆ $cmdLineOptionMap

RecompressTracked::$cmdLineOptionMap
staticprivate
Initial value:
= [
'no-count' => 'noCount'

Definition at line 82 of file recompressTracked.php.

◆ $copyOnly

RecompressTracked::$copyOnly = false

Definition at line 67 of file recompressTracked.php.

◆ $criticalLog

RecompressTracked::$criticalLog

Definition at line 71 of file recompressTracked.php.

◆ $debugLog

RecompressTracked::$debugLog

Definition at line 71 of file recompressTracked.php.

◆ $destClusters

RecompressTracked::$destClusters

Definition at line 59 of file recompressTracked.php.

◆ $infoLog

RecompressTracked::$infoLog

Definition at line 71 of file recompressTracked.php.

◆ $isChild

RecompressTracked::$isChild = false

Definition at line 68 of file recompressTracked.php.

◆ $noCount

RecompressTracked::$noCount = false

Definition at line 70 of file recompressTracked.php.

◆ $numBatches

RecompressTracked::$numBatches = 0

Definition at line 64 of file recompressTracked.php.

◆ $numProcs

RecompressTracked::$numProcs = 1

Definition at line 63 of file recompressTracked.php.

Referenced by dispatch(), killReplicaProcs(), and startReplicaProcs().

◆ $optionsWithArgs

RecompressTracked::$optionsWithArgs
staticprivate
Initial value:
= [
'procs',
'replica-id',
'debug-log',
'info-log',
'critical-log'
]

Definition at line 74 of file recompressTracked.php.

◆ $orphanBatchSize

RecompressTracked::$orphanBatchSize = 1000

Definition at line 61 of file recompressTracked.php.

Referenced by doAllOrphans().

◆ $orphanBlobClass

RecompressTracked::$orphanBlobClass

Definition at line 65 of file recompressTracked.php.

◆ $pageBlobClass

RecompressTracked::$pageBlobClass

Definition at line 65 of file recompressTracked.php.

◆ $prevReplicaId

RecompressTracked::$prevReplicaId

Definition at line 66 of file recompressTracked.php.

◆ $replicaId

RecompressTracked::$replicaId = false

Definition at line 69 of file recompressTracked.php.

Referenced by dispatch(), and dispatchToReplica().

◆ $replicaPipes

RecompressTracked::$replicaPipes

Definition at line 66 of file recompressTracked.php.

Referenced by dispatch().

◆ $replicaProcs

RecompressTracked::$replicaProcs

Definition at line 66 of file recompressTracked.php.

◆ $reportingInterval

RecompressTracked::$reportingInterval = 10

Definition at line 62 of file recompressTracked.php.

◆ $store

RecompressTracked::$store

Definition at line 72 of file recompressTracked.php.

Referenced by CgzCopyTransaction\commit().


The documentation for this class was generated from the following file: