Code Coverage |
||||||||||
Lines |
Functions and Methods |
Classes and Traits |
||||||||
Total | |
67.00% |
67 / 100 |
|
76.92% |
10 / 13 |
CRAP | |
0.00% |
0 / 1 |
DeferredUpdates | |
67.68% |
67 / 99 |
|
76.92% |
10 / 13 |
79.77 | |
0.00% |
0 / 1 |
getScopeStack | |
100.00% |
2 / 2 |
|
100.00% |
1 / 1 |
1 | |||
setScopeStack | |
0.00% |
0 / 3 |
|
0.00% |
0 / 1 |
6 | |||
addUpdate | |
100.00% |
2 / 2 |
|
100.00% |
1 / 1 |
1 | |||
addCallableUpdate | |
100.00% |
1 / 1 |
|
100.00% |
1 / 1 |
1 | |||
run | |
41.67% |
15 / 36 |
|
0.00% |
0 / 1 |
13.15 | |||
doUpdates | |
78.38% |
29 / 37 |
|
0.00% |
0 / 1 |
17.27 | |||
tryOpportunisticExecute | |
100.00% |
7 / 7 |
|
100.00% |
1 / 1 |
4 | |||
preventOpportunisticUpdates | |
100.00% |
4 / 4 |
|
100.00% |
1 / 1 |
1 | |||
pendingUpdatesCount | |
100.00% |
1 / 1 |
|
100.00% |
1 / 1 |
1 | |||
getPendingUpdates | |
100.00% |
1 / 1 |
|
100.00% |
1 / 1 |
1 | |||
clearPendingUpdates | |
100.00% |
1 / 1 |
|
100.00% |
1 / 1 |
1 | |||
getRecursiveExecutionStackDepth | |
100.00% |
1 / 1 |
|
100.00% |
1 / 1 |
1 | |||
attemptUpdate | |
100.00% |
3 / 3 |
|
100.00% |
1 / 1 |
1 |
1 | <?php |
2 | /** |
3 | * This program is free software; you can redistribute it and/or modify |
4 | * it under the terms of the GNU General Public License as published by |
5 | * the Free Software Foundation; either version 2 of the License, or |
6 | * (at your option) any later version. |
7 | * |
8 | * This program is distributed in the hope that it will be useful, |
9 | * but WITHOUT ANY WARRANTY; without even the implied warranty of |
10 | * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
11 | * GNU General Public License for more details. |
12 | * |
13 | * You should have received a copy of the GNU General Public License along |
14 | * with this program; if not, write to the Free Software Foundation, Inc., |
15 | * 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. |
16 | * http://www.gnu.org/copyleft/gpl.html |
17 | * |
18 | * @file |
19 | */ |
20 | |
21 | namespace MediaWiki\Deferred; |
22 | |
23 | use ErrorPageError; |
24 | use LogicException; |
25 | use MediaWiki\Logger\LoggerFactory; |
26 | use MWExceptionHandler; |
27 | use Throwable; |
28 | use Wikimedia\Rdbms\IDatabase; |
29 | use Wikimedia\ScopedCallback; |
30 | |
31 | /** |
32 | * Defer callable updates to run later in the PHP process |
33 | * |
34 | * This is a performance feature that enables MediaWiki to produce faster web responses. |
35 | * It allows you to postpone non-blocking work (e.g. work that does not change the web |
36 | * response) to after the HTTP response has been sent to the client (i.e. web browser). |
37 | * |
38 | * Once the response is finalized and sent to the browser, the webserver process stays |
39 | * for a little while longer (detached from the web request) to run your POSTSEND tasks. |
40 | * |
41 | * There is also a PRESEND option, which runs your task right before the finalized response |
42 | * is sent to the browser. This is for critical tasks that does need to block the response, |
43 | * but where you'd like to benefit from other DeferredUpdates features. Such as: |
44 | * |
45 | * - MergeableUpdate: batch updates from different components without coupling |
46 | * or awareness of each other. |
47 | * - Automatic cancellation: pass a IDatabase object (for any wiki or database) to |
48 | * DeferredUpdates::addCallableUpdate or AtomicSectionUpdate. |
49 | * - Reducing lock contention: if the response is likely to take several seconds |
50 | * (e.g. uploading a large file to FileBackend, or saving an edit to a large article) |
51 | * much of that work may overlap with a database transaction that is staying open for |
52 | * the entire duration. By moving contentious writes out to a PRESEND update, these |
53 | * get their own transaction (after the main one is committed), which give up some |
54 | * atomicity for improved throughput. |
55 | * |
56 | * ## Expectation and comparison to job queue |
57 | * |
58 | * When scheduling a POSTSEND via the DeferredUpdates system you can generally expect |
59 | * it to complete well before the client makes their next request. Updates runs directly after |
60 | * the web response is sent, from the same process on the same server. This unlike the JobQueue, |
61 | * where jobs may need to wait in line for some minutes or hours. |
62 | * |
63 | * If your update fails, this failure is not known to the client and gets no retry. For updates |
64 | * that need re-tries for system consistency or data integrity, it is recommended to implement |
65 | * it as a job instead and use JobQueueGroup::lazyPush. This has the caveat of being delayed |
66 | * by default, the same as any other job. |
67 | * |
68 | * A hybrid solution is available via the EnqueueableDataUpdate interface. By implementing |
69 | * this interface, you can queue your update via the DeferredUpdates first, and if it fails, |
70 | * the system will automatically catch this and queue it as a job instead. |
71 | * |
72 | * ## How it works during web requests |
73 | * |
74 | * 1. Your request route is executed (e.g. Action or SpecialPage class, or API). |
75 | * 2. Output is finalized and main database transaction is committed. |
76 | * 3. PRESEND updates run via DeferredUpdates::doUpdates. |
77 | * 5. The web response is sent to the browser. |
78 | * 6. POSTSEND updates run via DeferredUpdates::doUpdates. |
79 | * |
80 | * @see MediaWiki::preOutputCommit |
81 | * @see MediaWiki::restInPeace |
82 | * |
83 | * ## How it works for Maintenance scripts |
84 | * |
85 | * In CLI mode, no distinction is made between PRESEND and POSTSEND deferred updates, |
86 | * and the queue is periodically executed throughout the process. |
87 | * |
88 | * @see DeferredUpdates::tryOpportunisticExecute |
89 | * |
90 | * ## How it works internally |
91 | * |
92 | * Each update is added via DeferredUpdates::addUpdate and stored in either the PRESEND or |
93 | * POSTSEND queue. If an update gets queued while another update is already running, then |
94 | * we store in a "sub"-queue associated with the current update. This allows nested updates |
95 | * to be completed before other updates, which improves ordering for process caching. |
96 | * |
97 | * @since 1.19 |
98 | */ |
99 | class DeferredUpdates { |
100 | /** @var int Process all updates; in web requests, use only after flushing output buffer */ |
101 | public const ALL = 0; |
102 | /** @var int Specify/process updates that should run before flushing output buffer */ |
103 | public const PRESEND = 1; |
104 | /** @var int Specify/process updates that should run after flushing output buffer */ |
105 | public const POSTSEND = 2; |
106 | |
107 | /** @var int[] List of "defer until" queue stages that can be reached */ |
108 | public const STAGES = [ self::PRESEND, self::POSTSEND ]; |
109 | |
110 | /** @var DeferredUpdatesScopeStack|null Queue states based on recursion level */ |
111 | private static $scopeStack; |
112 | |
113 | /** |
114 | * @var int Nesting level for preventOpportunisticUpdates() |
115 | */ |
116 | private static $preventOpportunisticUpdates = 0; |
117 | |
118 | /** |
119 | * @return DeferredUpdatesScopeStack |
120 | */ |
121 | private static function getScopeStack(): DeferredUpdatesScopeStack { |
122 | self::$scopeStack ??= new DeferredUpdatesScopeMediaWikiStack(); |
123 | return self::$scopeStack; |
124 | } |
125 | |
126 | /** |
127 | * @param DeferredUpdatesScopeStack $scopeStack |
128 | * @internal Only for use in tests. |
129 | */ |
130 | public static function setScopeStack( DeferredUpdatesScopeStack $scopeStack ): void { |
131 | if ( !defined( 'MW_PHPUNIT_TEST' ) ) { |
132 | throw new LogicException( 'Cannot reconfigure DeferredUpdates outside tests' ); |
133 | } |
134 | self::$scopeStack = $scopeStack; |
135 | } |
136 | |
137 | /** |
138 | * Add an update to the pending update queue for execution at the appropriate time |
139 | * |
140 | * In CLI mode, callback magic will also be used to run updates when safe |
141 | * |
142 | * If an update is already in progress, then what happens to this update is as follows: |
143 | * - If it has a "defer until" stage at/before the actual run stage of the innermost |
144 | * in-progress update, then it will go into the sub-queue of that in-progress update. |
145 | * As soon as that update completes, MergeableUpdate instances in its sub-queue will be |
146 | * merged into the top-queues and the non-MergeableUpdate instances will be executed. |
147 | * This is done to better isolate updates from the failures of other updates and reduce |
148 | * the chance of race conditions caused by updates not fully seeing the intended changes |
149 | * of previously enqueued and executed updates. |
150 | * - If it has a "defer until" stage later than the actual run stage of the innermost |
151 | * in-progress update, then it will go into the normal top-queue for that stage. |
152 | * |
153 | * @param DeferrableUpdate $update Some object that implements doUpdate() |
154 | * @param int $stage One of (DeferredUpdates::PRESEND, DeferredUpdates::POSTSEND) |
155 | * @since 1.28 Added the $stage parameter |
156 | */ |
157 | public static function addUpdate( DeferrableUpdate $update, $stage = self::POSTSEND ) { |
158 | self::getScopeStack()->current()->addUpdate( $update, $stage ); |
159 | self::tryOpportunisticExecute(); |
160 | } |
161 | |
162 | /** |
163 | * Add an update to the pending update queue that invokes the specified callback when run |
164 | * |
165 | * @param callable $callable One of the following: |
166 | * - A Closure callback that takes the caller name as its argument |
167 | * - A non-Closure callback that takes no arguments |
168 | * @param int $stage One of (DeferredUpdates::PRESEND, DeferredUpdates::POSTSEND) |
169 | * @param IDatabase|IDatabase[] $dependeeDbws DB handles which might have pending writes |
170 | * upon which this update depends. If any of the handles already has an open transaction, |
171 | * a rollback thereof will cause this update to be cancelled (if it has not already run). |
172 | * [optional] (since 1.28) |
173 | * @since 1.27 Added $stage parameter |
174 | * @since 1.28 Added the $dbw parameter |
175 | * @since 1.43 Closures are now given the caller name parameter |
176 | */ |
177 | public static function addCallableUpdate( |
178 | $callable, |
179 | $stage = self::POSTSEND, |
180 | $dependeeDbws = [] |
181 | ) { |
182 | self::addUpdate( new MWCallableUpdate( $callable, wfGetCaller(), $dependeeDbws ), $stage ); |
183 | } |
184 | |
185 | /** |
186 | * Run an update, and, if an error was thrown, catch/log it and enqueue the update as |
187 | * a job in the job queue system if possible (e.g. implements EnqueueableDataUpdate) |
188 | * |
189 | * @param DeferrableUpdate $update |
190 | * @return Throwable|null |
191 | */ |
192 | private static function run( DeferrableUpdate $update ): ?Throwable { |
193 | $logger = LoggerFactory::getInstance( 'DeferredUpdates' ); |
194 | |
195 | $type = get_class( $update ) |
196 | . ( $update instanceof DeferrableCallback ? '_' . $update->getOrigin() : '' ); |
197 | $updateId = spl_object_id( $update ); |
198 | $logger->debug( "DeferredUpdates::run: started $type #{updateId}", [ 'updateId' => $updateId ] ); |
199 | |
200 | $updateException = null; |
201 | |
202 | $startTime = microtime( true ); |
203 | try { |
204 | self::attemptUpdate( $update ); |
205 | } catch ( Throwable $updateException ) { |
206 | MWExceptionHandler::logException( $updateException ); |
207 | $logger->error( |
208 | "Deferred update '{deferred_type}' failed to run.", |
209 | [ |
210 | 'deferred_type' => $type, |
211 | 'exception' => $updateException, |
212 | ] |
213 | ); |
214 | self::getScopeStack()->onRunUpdateFailed( $update ); |
215 | } finally { |
216 | $walltime = microtime( true ) - $startTime; |
217 | $logger->debug( "DeferredUpdates::run: ended $type #{updateId}, processing time: {walltime}", [ |
218 | 'updateId' => $updateId, |
219 | 'walltime' => $walltime, |
220 | ] ); |
221 | } |
222 | |
223 | // Try to push the update as a job so it can run later if possible |
224 | if ( $updateException && $update instanceof EnqueueableDataUpdate ) { |
225 | try { |
226 | self::getScopeStack()->queueDataUpdate( $update ); |
227 | } catch ( Throwable $jobException ) { |
228 | MWExceptionHandler::logException( $jobException ); |
229 | $logger->error( |
230 | "Deferred update '{deferred_type}' failed to enqueue as a job.", |
231 | [ |
232 | 'deferred_type' => $type, |
233 | 'exception' => $jobException, |
234 | ] |
235 | ); |
236 | self::getScopeStack()->onRunUpdateFailed( $update ); |
237 | } |
238 | } |
239 | |
240 | return $updateException; |
241 | } |
242 | |
243 | /** |
244 | * Consume and execute all pending updates |
245 | * |
246 | * Note that it is rarely the case that this method should be called outside of a few |
247 | * select entry points. For simplicity, that kind of recursion is discouraged. Recursion |
248 | * cannot happen if an explicit transaction round is active, which limits usage to updates |
249 | * with TRX_ROUND_ABSENT that do not leave open any transactions round of their own during |
250 | * the call to this method. |
251 | * |
252 | * In the less-common case of this being called within an in-progress DeferrableUpdate, |
253 | * this will not see any top-queue updates (since they were consumed and are being run |
254 | * inside an outer execution loop). In that case, it will instead operate on the sub-queue |
255 | * of the innermost in-progress update on the stack. |
256 | * |
257 | * @internal For use by MediaWiki, Maintenance, JobRunner, JobExecutor |
258 | * @param int $stage Which updates to process. One of |
259 | * (DeferredUpdates::PRESEND, DeferredUpdates::POSTSEND, DeferredUpdates::ALL) |
260 | */ |
261 | public static function doUpdates( $stage = self::ALL ) { |
262 | /** @var ErrorPageError $guiError First presentable client-level error thrown */ |
263 | $guiError = null; |
264 | /** @var Throwable $exception First of any error thrown */ |
265 | $exception = null; |
266 | |
267 | $scope = self::getScopeStack()->current(); |
268 | |
269 | // T249069: recursion is not possible once explicit transaction rounds are involved |
270 | $activeUpdate = $scope->getActiveUpdate(); |
271 | if ( $activeUpdate ) { |
272 | $class = get_class( $activeUpdate ); |
273 | if ( !( $activeUpdate instanceof TransactionRoundAwareUpdate ) ) { |
274 | throw new LogicException( |
275 | __METHOD__ . ": reached from $class, which is not TransactionRoundAwareUpdate" |
276 | ); |
277 | } |
278 | if ( $activeUpdate->getTransactionRoundRequirement() !== $activeUpdate::TRX_ROUND_ABSENT ) { |
279 | throw new LogicException( |
280 | __METHOD__ . ": reached from $class, which does not specify TRX_ROUND_ABSENT" |
281 | ); |
282 | } |
283 | } |
284 | |
285 | $scope->processUpdates( |
286 | $stage, |
287 | static function ( DeferrableUpdate $update, $activeStage ) use ( &$guiError, &$exception ) { |
288 | $scopeStack = self::getScopeStack(); |
289 | $childScope = $scopeStack->descend( $activeStage, $update ); |
290 | try { |
291 | $e = self::run( $update ); |
292 | $guiError = $guiError ?: ( $e instanceof ErrorPageError ? $e : null ); |
293 | $exception = $exception ?: $e; |
294 | // Any addUpdate() calls between descend() and ascend() used the sub-queue. |
295 | // In rare cases, DeferrableUpdate::doUpdates() will process them by calling |
296 | // doUpdates() itself. In any case, process remaining updates in the subqueue. |
297 | // them, enqueueing them, or transferring them to the parent scope |
298 | // queues as appropriate... |
299 | $childScope->processUpdates( |
300 | $activeStage, |
301 | static function ( DeferrableUpdate $sub ) use ( &$guiError, &$exception ) { |
302 | $e = self::run( $sub ); |
303 | $guiError = $guiError ?: ( $e instanceof ErrorPageError ? $e : null ); |
304 | $exception = $exception ?: $e; |
305 | } |
306 | ); |
307 | } finally { |
308 | $scopeStack->ascend(); |
309 | } |
310 | } |
311 | ); |
312 | |
313 | // VW-style hack to work around T190178, so we can make sure |
314 | // PageMetaDataUpdater doesn't throw exceptions. |
315 | if ( $exception && defined( 'MW_PHPUNIT_TEST' ) ) { |
316 | throw $exception; |
317 | } |
318 | |
319 | // Throw the first of any GUI errors as long as the context is HTTP pre-send. However, |
320 | // callers should check permissions *before* enqueueing updates. If the main transaction |
321 | // round actions succeed but some deferred updates fail due to permissions errors then |
322 | // there is a risk that some secondary data was not properly updated. |
323 | if ( $guiError && $stage === self::PRESEND && !headers_sent() ) { |
324 | throw $guiError; |
325 | } |
326 | } |
327 | |
328 | /** |
329 | * Consume and execute pending updates now if possible, instead of waiting. |
330 | * |
331 | * In web requests, updates are always deferred until the end of the request. |
332 | * |
333 | * In CLI mode, updates run earlier and more often. This is important for long-running |
334 | * Maintenance scripts that would otherwise grow an excessively large queue, which increases |
335 | * memory use, and risks losing all updates if the script ends early or crashes. |
336 | * |
337 | * The folllowing conditions are required for updates to run early in CLI mode: |
338 | * |
339 | * - No update is already in progress (ensure linear flow, recursion guard). |
340 | * - LBFactory indicates that we don't have any "busy" database connections, i.e. |
341 | * there are no pending writes or otherwise active and uncommitted transactions, |
342 | * except if the transaction is empty and merely used for primary DB read queries, |
343 | * in which case the transaction (and its repeatable-read snapshot) can be safely flushed. |
344 | * |
345 | * How this works: |
346 | * |
347 | * - When a maintenance script commits a change or waits for replication, such as |
348 | * via IConnectionProvider::commitAndWaitForReplication, then ILBFactory calls |
349 | * tryOpportunisticExecute(). This is injected via MWLBFactory::applyGlobalState. |
350 | * |
351 | * - For maintenance scripts that don't do much with the database, we also call |
352 | * tryOpportunisticExecute() after every addUpdate() call. |
353 | * |
354 | * - Upon the completion of Maintenance::execute() via Maintenance::shutdown(), |
355 | * any remaining updates are run. |
356 | * |
357 | * Note that this method runs both PRESEND and POSTSEND updates and thus should not be called |
358 | * during web requests. It is only intended for long-running Maintenance scripts. |
359 | * |
360 | * @internal For use by Maintenance |
361 | * @since 1.28 |
362 | * @return bool Whether updates were allowed to run |
363 | */ |
364 | public static function tryOpportunisticExecute(): bool { |
365 | // Leave execution up to the current loop if an update is already in progress |
366 | // or if updates are explicitly disabled |
367 | if ( self::getRecursiveExecutionStackDepth() |
368 | || self::$preventOpportunisticUpdates |
369 | ) { |
370 | return false; |
371 | } |
372 | |
373 | if ( self::getScopeStack()->allowOpportunisticUpdates() ) { |
374 | self::doUpdates( self::ALL ); |
375 | return true; |
376 | } |
377 | |
378 | return false; |
379 | } |
380 | |
381 | /** |
382 | * Prevent opportunistic updates until the returned ScopedCallback is |
383 | * consumed. |
384 | * |
385 | * @return ScopedCallback |
386 | */ |
387 | public static function preventOpportunisticUpdates() { |
388 | self::$preventOpportunisticUpdates++; |
389 | return new ScopedCallback( static function () { |
390 | self::$preventOpportunisticUpdates--; |
391 | } ); |
392 | } |
393 | |
394 | /** |
395 | * Get the number of pending updates for the current execution context |
396 | * |
397 | * If an update is in progress, then this operates on the sub-queues of the |
398 | * innermost in-progress update. Otherwise, it acts on the top-queues. |
399 | * |
400 | * @return int |
401 | * @since 1.28 |
402 | */ |
403 | public static function pendingUpdatesCount() { |
404 | return self::getScopeStack()->current()->pendingUpdatesCount(); |
405 | } |
406 | |
407 | /** |
408 | * Get a list of the pending updates for the current execution context |
409 | * |
410 | * If an update is in progress, then this operates on the sub-queues of the |
411 | * innermost in-progress update. Otherwise, it acts on the top-queues. |
412 | * |
413 | * @param int $stage Look for updates with this "defer until" stage. One of |
414 | * (DeferredUpdates::PRESEND, DeferredUpdates::POSTSEND, DeferredUpdates::ALL) |
415 | * @return DeferrableUpdate[] |
416 | * @internal This method should only be used for unit tests |
417 | * @since 1.29 |
418 | */ |
419 | public static function getPendingUpdates( $stage = self::ALL ) { |
420 | return self::getScopeStack()->current()->getPendingUpdates( $stage ); |
421 | } |
422 | |
423 | /** |
424 | * Cancel all pending updates for the current execution context |
425 | * |
426 | * If an update is in progress, then this operates on the sub-queues of the |
427 | * innermost in-progress update. Otherwise, it acts on the top-queues. |
428 | * |
429 | * @internal This method should only be used for unit tests |
430 | */ |
431 | public static function clearPendingUpdates() { |
432 | self::getScopeStack()->current()->clearPendingUpdates(); |
433 | } |
434 | |
435 | /** |
436 | * Get the number of in-progress calls to DeferredUpdates::doUpdates() |
437 | * |
438 | * @return int |
439 | * @internal This method should only be used for unit tests |
440 | */ |
441 | public static function getRecursiveExecutionStackDepth() { |
442 | return self::getScopeStack()->getRecursiveDepth(); |
443 | } |
444 | |
445 | /** |
446 | * Attempt to run an update with the appropriate transaction round state if needed |
447 | * |
448 | * It is allowed for a DeferredUpdate to directly execute one or more other DeferredUpdate |
449 | * instances without queueing them by calling this method. In that case, the outer update |
450 | * must use TransactionRoundAwareUpdate::TRX_ROUND_ABSENT, e.g. by extending |
451 | * TransactionRoundDefiningUpdate, so that this method can give each update its own |
452 | * transaction round. |
453 | * |
454 | * @param DeferrableUpdate $update |
455 | * @since 1.34 |
456 | */ |
457 | public static function attemptUpdate( DeferrableUpdate $update ) { |
458 | self::getScopeStack()->onRunUpdateStart( $update ); |
459 | |
460 | $update->doUpdate(); |
461 | |
462 | self::getScopeStack()->onRunUpdateEnd( $update ); |
463 | } |
464 | } |
465 | |
466 | /** @deprecated class alias since 1.42 */ |
467 | class_alias( DeferredUpdates::class, 'DeferredUpdates' ); |