MediaWiki REL1_32
cleanupInvalidDbKeys.php
Go to the documentation of this file.
1<?php
24require_once __DIR__ . '/Maintenance.php';
25
34 protected static $tables = [
35 // Data tables
36 [ 'page', 'page' ],
37 [ 'redirect', 'rd', 'idField' => 'rd_from' ],
38 [ 'archive', 'ar' ],
39 [ 'logging', 'log' ],
40 [ 'protected_titles', 'pt', 'idField' => 0 ],
41 [ 'category', 'cat', 'nsField' => 14 ],
42 [ 'recentchanges', 'rc' ],
43 [ 'watchlist', 'wl' ],
44 // The querycache tables' qc(c)_title and qcc_titletwo may contain titles,
45 // but also usernames or other things like that, so we leave them alone
46
47 // Links tables
48 [ 'pagelinks', 'pl', 'idField' => 'pl_from' ],
49 [ 'templatelinks', 'tl', 'idField' => 'tl_from' ],
50 [ 'categorylinks', 'cl', 'idField' => 'cl_from', 'nsField' => 14, 'titleField' => 'cl_to' ],
51 ];
52
53 public function __construct() {
54 parent::__construct();
55 $this->addDescription( <<<'TEXT'
56This script cleans up the title fields in various tables to remove entries that
57will be rejected by the constructor of TitleValue. This constructor throws an
58exception when invalid data is encountered, which will not normally occur on
59regular page views, but can happen on query special pages.
60
61The script targets titles matching the regular expression /^_|[ \r\n\t]|_$/.
62Because any foreign key relationships involving these titles will already be
63broken, the titles are corrected to a valid version or the rows are deleted
64entirely, depending on the table.
65
66The script runs with the expectation that STDOUT is redirected to a file.
67TEXT
68 );
69 $this->addOption( 'fix', 'Actually clean up invalid titles. If this parameter is ' .
70 'not specified, the script will report invalid titles but not clean them up.',
71 false, false );
72 $this->addOption( 'table', 'The table(s) to process. This option can be specified ' .
73 'more than once (e.g. -t category -t watchlist). If not specified, all available ' .
74 'tables will be processed. Available tables are: ' .
75 implode( ', ', array_column( static::$tables, 0 ) ), false, true, 't', true );
76
77 $this->setBatchSize( 500 );
78 }
79
80 public function execute() {
81 $tablesToProcess = $this->getOption( 'table' );
82 foreach ( static::$tables as $tableParams ) {
83 if ( !$tablesToProcess || in_array( $tableParams[0], $tablesToProcess ) ) {
84 $this->cleanupTable( $tableParams );
85 }
86 }
87
88 $this->outputStatus( 'Done!' );
89 if ( $this->hasOption( 'fix' ) ) {
90 $this->outputStatus( ' Cleaned up invalid DB keys on ' . wfWikiID() . "!\n" );
91 }
92 }
93
101 protected function outputStatus( $str, $channel = null ) {
102 // Make it easier to find progress lines in the STDOUT log
103 if ( trim( $str ) ) {
104 fwrite( STDOUT, '*** ' . trim( $str ) . "\n" );
105 }
106 fwrite( STDERR, $str );
107 }
108
114 protected function writeToReport( $str ) {
115 fwrite( STDOUT, $str );
116 }
117
123 protected function cleanupTable( $tableParams ) {
124 $table = $tableParams[0];
125 $prefix = $tableParams[1];
126 $idField = $tableParams['idField'] ?? "{$prefix}_id";
127 $nsField = $tableParams['nsField'] ?? "{$prefix}_namespace";
128 $titleField = $tableParams['titleField'] ?? "{$prefix}_title";
129
130 $this->outputStatus( "Looking for invalid $titleField entries in $table...\n" );
131
132 // Do all the select queries on the replicas, as they are slow (they use
133 // unanchored LIKEs). Naturally this could cause problems if rows are
134 // modified after selecting and before deleting/updating, but working on
135 // the hypothesis that invalid rows will be old and in all likelihood
136 // unreferenced, we should be fine to do it like this.
137 $dbr = $this->getDB( DB_REPLICA, 'vslow' );
138
139 // Find all TitleValue-invalid titles.
140 $percent = $dbr->anyString(); // DBMS-agnostic equivalent of '%' LIKE wildcard
141 $res = $dbr->select(
142 $table,
143 [
144 'id' => $idField,
145 'ns' => $nsField,
146 'title' => $titleField,
147 ],
148 // The REGEXP operator is not cross-DBMS, so we have to use lots of LIKEs
149 [ $dbr->makeList( [
150 $titleField . $dbr->buildLike( $percent, ' ', $percent ),
151 $titleField . $dbr->buildLike( $percent, "\r", $percent ),
152 $titleField . $dbr->buildLike( $percent, "\n", $percent ),
153 $titleField . $dbr->buildLike( $percent, "\t", $percent ),
154 $titleField . $dbr->buildLike( '_', $percent ),
155 $titleField . $dbr->buildLike( $percent, '_' ),
156 ], LIST_OR ) ],
157 __METHOD__,
158 [ 'LIMIT' => $this->getBatchSize() ]
159 );
160
161 $this->outputStatus( "Number of invalid rows: " . $res->numRows() . "\n" );
162 if ( !$res->numRows() ) {
163 $this->outputStatus( "\n" );
164 return;
165 }
166
167 // Write a table of titles to the report file. Also keep a list of the found
168 // IDs, as we might need it later for DB updates
169 $this->writeToReport( sprintf( "%10s | ns | dbkey\n", $idField ) );
170 $ids = [];
171 foreach ( $res as $row ) {
172 $this->writeToReport( sprintf( "%10d | %3d | %s\n", $row->id, $row->ns, $row->title ) );
173 $ids[] = $row->id;
174 }
175
176 // If we're doing a dry run, output the new titles we would use for the UPDATE
177 // queries (if relevant), and finish
178 if ( !$this->hasOption( 'fix' ) ) {
179 if ( $table === 'logging' || $table === 'archive' ) {
180 $this->writeToReport( "The following updates would be run with the --fix flag:\n" );
181 foreach ( $res as $row ) {
182 $newTitle = self::makeValidTitle( $row->title );
183 $this->writeToReport(
184 "$idField={$row->id}: update '{$row->title}' to '$newTitle'\n" );
185 }
186 }
187
188 if ( $table !== 'page' && $table !== 'redirect' ) {
189 $this->outputStatus( "Run with --fix to clean up these rows\n" );
190 }
191 $this->outputStatus( "\n" );
192 return;
193 }
194
195 // Fix the bad data, using different logic for the various tables
196 $dbw = $this->getDB( DB_MASTER );
197 switch ( $table ) {
198 case 'page':
199 case 'redirect':
200 // This shouldn't happen on production wikis, and we already have a script
201 // to handle 'page' rows anyway, so just notify the user and let them decide
202 // what to do next.
203 $this->outputStatus( <<<TEXT
204IMPORTANT: This script does not fix invalid entries in the $table table.
205Consider repairing these rows, and rows in related tables, by hand.
206You may like to run, or borrow logic from, the cleanupTitles.php script.
207
208TEXT
209 );
210 break;
211
212 case 'archive':
213 case 'logging':
214 // Rename the title to a corrected equivalent. Any foreign key relationships
215 // to the page_title field are already broken, so this will just make sure
216 // users can still access the log entries/deleted revisions from the interface
217 // using a valid page title.
218 $this->outputStatus(
219 "Updating these rows, setting $titleField to the closest valid DB key...\n" );
220 $affectedRowCount = 0;
221 foreach ( $res as $row ) {
222 $newTitle = self::makeValidTitle( $row->title );
223 $this->writeToReport(
224 "$idField={$row->id}: updating '{$row->title}' to '$newTitle'\n" );
225
226 $dbw->update( $table,
227 [ $titleField => $newTitle ],
228 [ $idField => $row->id ],
229 __METHOD__ );
230 $affectedRowCount += $dbw->affectedRows();
231 }
233 $this->outputStatus( "Updated $affectedRowCount rows on $table.\n" );
234
235 break;
236
237 case 'recentchanges':
238 case 'watchlist':
239 case 'category':
240 // Since these broken titles can't exist, there's really nothing to watch,
241 // nothing can be categorised in them, and they can't have been changed
242 // recently, so we can just remove these rows.
243 $this->outputStatus( "Deleting invalid $table rows...\n" );
244 $dbw->delete( $table, [ $idField => $ids ], __METHOD__ );
246 $this->outputStatus( 'Deleted ' . $dbw->affectedRows() . " rows from $table.\n" );
247 break;
248
249 case 'protected_titles':
250 // Since these broken titles can't exist, there's really nothing to protect,
251 // so we can just remove these rows. Made more complicated by this table
252 // not having an ID field
253 $this->outputStatus( "Deleting invalid $table rows...\n" );
254 $affectedRowCount = 0;
255 foreach ( $res as $row ) {
256 $dbw->delete( $table,
257 [ $nsField => $row->ns, $titleField => $row->title ],
258 __METHOD__ );
259 $affectedRowCount += $dbw->affectedRows();
260 }
262 $this->outputStatus( "Deleted $affectedRowCount rows from $table.\n" );
263 break;
264
265 case 'pagelinks':
266 case 'templatelinks':
267 case 'categorylinks':
268 // Update links tables for each page where these bogus links are supposedly
269 // located. If the invalid rows don't go away after these jobs go through,
270 // they're probably being added by a buggy hook.
271 $this->outputStatus( "Queueing link update jobs for the pages in $idField...\n" );
272 foreach ( $res as $row ) {
273 $wp = WikiPage::newFromID( $row->id );
274 if ( $wp ) {
276 } else {
277 // This link entry points to a nonexistent page, so just get rid of it
278 $dbw->delete( $table,
279 [ $idField => $row->id, $nsField => $row->ns, $titleField => $row->title ],
280 __METHOD__ );
281 }
282 }
284 $this->outputStatus( "Link update jobs have been added to the job queue.\n" );
285 break;
286 }
287
288 $this->outputStatus( "\n" );
289 }
290
297 protected static function makeValidTitle( $invalidTitle ) {
298 return strtr( trim( $invalidTitle, '_' ),
299 [ ' ' => '_', "\r" => '', "\n" => '', "\t" => '_' ] );
300 }
301}
302
303$maintClass = CleanupInvalidDbKeys::class;
304require_once RUN_MAINTENANCE_IF_MAIN;
and(b) You must cause any modified files to carry prominent notices stating that You changed the files
Apache License January AND DISTRIBUTION Definitions License shall mean the terms and conditions for and distribution as defined by Sections through of this document Licensor shall mean the copyright owner or entity authorized by the copyright owner that is granting the License Legal Entity shall mean the union of the acting entity and all other entities that control are controlled by or are under common control with that entity For the purposes of this definition control direct or to cause the direction or management of such whether by contract or including but not limited to software source documentation and configuration files Object form shall mean any form resulting from mechanical transformation or translation of a Source including but not limited to compiled object generated and conversions to other media types Work shall mean the work of whether in Source or Object made available under the as indicated by a copyright notice that is included in or attached to the whether in Source or Object that is based or other modifications as a an original work of authorship For the purposes of this Derivative Works shall not include works that remain separable or merely the Work and Derivative Works thereof Contribution shall mean any work of including the original version of the Work and any modifications or additions to that Work or Derivative Works that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner For the purposes of this submitted means any form of or written communication sent to the Licensor or its including but not limited to communication on electronic mailing source code control and issue tracking systems that are managed by
Apache License January AND DISTRIBUTION Definitions License shall mean the terms and conditions for and distribution as defined by Sections through of this document Licensor shall mean the copyright owner or entity authorized by the copyright owner that is granting the License Legal Entity shall mean the union of the acting entity and all other entities that control are controlled by or are under common control with that entity For the purposes of this definition control direct or to cause the direction or management of such whether by contract or including but not limited to software source documentation and configuration files Object form shall mean any form resulting from mechanical transformation or translation of a Source including but not limited to compiled object generated and conversions to other media types Work shall mean the work of whether in Source or Object made available under the as indicated by a copyright notice that is included in or attached to the whether in Source or Object that is based or other modifications as a an original work of authorship For the purposes of this Derivative Works shall not include works that remain separable from
target page
</td >< td > &</td >< td > t want your writing to be edited mercilessly and redistributed at will
wfWaitForSlaves( $ifWritesSince=null, $wiki=false, $cluster=false, $timeout=null)
Waits for the replica DBs to catch up to the master position.
wfWikiID()
Get an ASCII string identifying this wiki This is used as a prefix in memcached keys.
Maintenance script that cleans up invalid titles in various tables.
static makeValidTitle( $invalidTitle)
Fix possible validation issues in the given title (DB key).
cleanupTable( $tableParams)
Identifies, and optionally cleans up, invalid titles.
outputStatus( $str, $channel=null)
Prints text to STDOUT, and STDERR if STDOUT was redirected to a file.
writeToReport( $str)
Prints text to STDOUT.
static array $tables
List of tables to clean up, and the field prefix for that table.
__construct()
Default constructor.
execute()
Do the actual work.
Abstract maintenance class for quickly writing and churning out maintenance scripts with minimal effo...
getDB( $db, $groups=[], $wiki=false)
Returns a database to be used by current maintenance script.
hasOption( $name)
Checks to see if a particular option exists.
addDescription( $text)
Set the description text.
addOption( $name, $description, $required=false, $withArg=false, $shortName=false, $multiOccurrence=false)
Add a parameter to the script.
getOption( $name, $default=null)
Get an option, or return the default.
setBatchSize( $s=0)
Set the batch size.
Represents a page (or page fragment) title within MediaWiki.
The ContentHandler facility adds support for arbitrary content types on wiki pages
The ContentHandler facility adds support for arbitrary content types on wiki instead of relying on wikitext for everything It was introduced in MediaWiki Each kind of and so on Built in content types are
$res
Definition database.txt:21
For a write use something like
Definition database.txt:28
For a write query
Definition database.txt:26
deferred txt A few of the database updates required by various functions here can be deferred until after the result page is displayed to the user For updating the view updating the linked to tables after a etc PHP does not yet have any way to tell the server to actually return and disconnect while still running these but it might have such a feature in the future We handle these by creating a deferred update object and putting those objects on a global then executing the whole list after the page is displayed We don t do anything smart like collating updates to the same table or such because the list is almost always going to have just one item on if that
Definition deferred.txt:13
deferred txt A few of the database updates required by various functions here can be deferred until after the result page is displayed to the user For updating the view updating the linked to tables after a etc PHP does not yet have any way to tell the server to actually return and disconnect while still running these but it might have such a feature in the future We handle these by creating a deferred update object and putting those objects on a global then executing the whole list after the page is displayed We don t do anything smart like collating updates to the same table or such because the list is almost always going to have just one item on if so it s not worth the trouble Since there is a job queue in the jobs table
Definition deferred.txt:16
This document is intended to provide useful advice for parties seeking to redistribute MediaWiki to end users It s targeted particularly at maintainers for Linux since it s been observed that distribution packages of MediaWiki often break We ve consistently had to recommend that users seeking support use official tarballs instead of their distribution s and this often solves whatever problem the user is having It would be nice if this could such as
globals txt Globals are evil The original MediaWiki code relied on globals for processing context far too often MediaWiki development since then has been a story of slowly moving context out of global variables and into objects Storing processing context in object member variables allows those objects to be reused in a much more flexible way Consider the elegance of
database rows
Definition globals.txt:10
globals will be eliminated from MediaWiki entirely
Definition globals.txt:29
const LIST_OR
Definition Defines.php:46
either a unescaped string or a HtmlArmor object after in associative array form externallinks including delete and has completed for all link tables whether this was an auto creation use $formDescriptor instead default is conds Array Extra conditions for the No matching items in log is displayed if loglist is empty msgKey Array If you want a nice box with a set this to the key of the message First element is the message key
Definition hooks.txt:2214
null for the local wiki Added in
Definition hooks.txt:1627
null means default in associative array with keys and values unescaped Should be merged with default with a value of false meaning to suppress the attribute in associative array with keys and values unescaped broken
Definition hooks.txt:2050
return true to allow those checks to occur
Definition hooks.txt:1515
Using a hook running we can avoid having all this option specific stuff in our mainline code Using the function We ve cleaned up the code here by removing clumps of infrequently used code and moving them off somewhere else It s much easier for someone working with this code to see what s _really_ going and make changes or fix bugs In we can take all the code that deals with the little used title reversing we can concentrate it all in an extension file
Definition hooks.txt:106
Using a hook running we can avoid having all this option specific stuff in our mainline code Using the function We ve cleaned up the code here by removing clumps of infrequently used code and moving them off somewhere else It s much easier for someone working with this code to see what s _really_ going on
Definition hooks.txt:86
injection txt This is an overview of how MediaWiki makes use of dependency injection The design described here grew from the discussion of RFC T384 The term dependency this means that anything an object needs to operate should be injected from the the object itself should only know narrow no concrete implementation of the logic it relies on The requirement to inject everything typically results in an architecture that based on two main types of and essentially stateless service objects that use other service objects to operate on the value objects As of the beginning MediaWiki is only starting to use the DI approach Much of the code still relies on global state or direct resulting in a highly cyclical dependency which acts as the top level factory for services in MediaWiki which can be used to gain access to default instances of various services MediaWikiServices however also allows new services to be defined and default services to be redefined Services are defined or redefined by providing a callback the instantiator that will return a new instance of the service When it will create an instance of MediaWikiServices and populate it with the services defined in the files listed by thereby bootstrapping the DI framework Per $wgServiceWiringFiles lists includes ServiceWiring php
Definition injection.txt:37
Prior to version
require_once RUN_MAINTENANCE_IF_MAIN
The wiki should then use memcached to cache various data To use multiple just add more items to the array To increase the weight of a make its entry a array("192.168.0.1:11211", 2))
This document provides an overview of the usage of PageUpdater and that is
In both all secondary updates will be triggered handle like object that caches derived data representing a and can trigger updates of cached copies of that e g in the links tables
title
This document describes the state of Postgres support in and is fairly well maintained The main code is very well while extensions are very hit and miss it is probably the most supported database after MySQL Much of the work in making MediaWiki database agnostic came about through the work of creating Postgres as and are nearing end of but without copying over all the usage comments General notes on the but these can almost always be programmed around *Although Postgres has a true BOOLEAN boolean columns are always mapped to as the code does not always treat the column as a and VARBINARY columns should simply be TEXT The only exception is when VARBINARY is used to store true binary data
Definition postgres.txt:43
const DB_REPLICA
Definition defines.php:25
const DB_MASTER
Definition defines.php:26