This class contains general utilities for: (a) querying token properties and token types (b) manipulating tokens, individually and as collections.
More...
|
| static | isWikitextBlockTag (string $name) |
| |
| static | tagOpensBlockScope (string $name) |
| | In the legacy parser, these block tags open block-tag scope See doBlockLevels in the PHP parser (includes/parser/Parser.php).
|
| |
| static | tagClosesBlockScope (string $name) |
| | In the legacy parser, these block tags close block-tag scope See doBlockLevels in the PHP parser (includes/parser/Parser.php).
|
| |
| static | isTemplateToken ( $token) |
| | Is this a template token?
|
| |
| static | isTemplateArgToken ( $token) |
| | Is this a template arg token?
|
| |
| static | isExtensionToken ( $token) |
| | Is this an extension token?
|
| |
| static | isHTMLTag ( $token) |
| | Determine whether the current token was an HTML tag in wikitext.
|
| |
| static | hasDOMFragmentType (Token $token) |
| | Is the token a DOMFragment type value?
|
| |
| static | isTableTag ( $token) |
| | Is the token a table tag?
|
| |
| static | isSolTransparentLinkTag ( $token) |
| | Determine if token is a transparent link tag.
|
| |
| static | isBehaviorSwitch (Env $env, $token) |
| | Does this token represent a behavior switch?
|
| |
| static | isSolTransparent (Env $env, $token) |
| | This should come close to matching WTUtils::emitsSolTransparentSingleLineWT, without the single line caveat.
|
| |
| static | isAnnotationMetaToken (Token $t) |
| |
| static | isAnnotationStartToken (Token $t) |
| | Checks whether the provided meta tag token is an annotation start token.
|
| |
| static | isAnnotationEndToken (Token $t) |
| | Checks whether the provided meta tag token is an annotation end token.
|
| |
| static | isTranslationUnitMarker (Env $env, CommentTk $token) |
| | HACK: Returns true if $token looks like a TU marker () and if we could be in a translate-annotated page.
|
| |
| static | matchTypeOf (Token $t, string $typeRe) |
| | Determine whether the token matches the given typeof attribute value.
|
| |
| static | hasTypeOf (Token $t, string $type) |
| | Determine whether the token matches the given typeof attribute value.
|
| |
| static | dedupeAboutIds (Env $env, array $maybeTokens) |
| |
| static | shiftTokenTSR (array $tokens, ?int $offset, ?Source $tsrSource=null) |
| | Shift TSR of a token by the requested $offset value and optionally, update its TSR source.
|
| |
|
static | resetSource (array $tokens, Source $tsrSource) |
| |
| static | stripEOFTkFromTokens (array &$tokens) |
| | Strip EOFTk token from token chunk.
|
| |
| static | convertOffsets (string $s, string $from, string $to, array $offsets) |
| | Convert string offsets.
|
| |
| static | convertTokenOffsets (string $s, string $from, string $to, array $tokens) |
| | Convert offsets in a token array.
|
| |
| static | isEntitySpanToken ( $token) |
| | Tests whether token represents an HTML entity.
|
| |
| static | newlinesToNlTks (string $str) |
| | Transform "\n" and "\r\n" in the input string to NlTk tokens.
|
| |
| static | tokensToString ( $tokens, bool $strict=false, array $opts=[], ?int $max=null) |
| | Flatten/convert a token array into a string.
|
| |
| static | kvToHash (array $kvs) |
| | Convert an array of key-value pairs into a hash of keys to values.
|
| |
| static | tokenTrim ( $tokens) |
| | Trim space and newlines from leading and trailing text tokens.
|
| |
| static | hasTemplateToken ( $tokens) |
| | Detect, if array (or any iterable container) contains template token.
|
| |
This class contains general utilities for: (a) querying token properties and token types (b) manipulating tokens, individually and as collections.
| static Wikimedia\Parsoid\Utils\TokenUtils::shiftTokenTSR |
( |
array | $tokens, |
|
|
?int | $offset, |
|
|
?Source | $tsrSource = null ) |
|
static |
Shift TSR of a token by the requested $offset value and optionally, update its TSR source.
At a basic level, "f(wt) = tokens" should be memoizable within the parser pipeline (since the config, env, etc. are fixed for the request) no matter where "wt" originated from (top-level or templates). But, embedded state like tsr offsets, and additional nested state like source ranges interfere with that memoizability. This method attempts to migrate over such embedded state reliably.
NOTE about $offset
A null value of $offset resets TSR on all tokens since we cannot compute a reliable new value of $tsr and the old value of $tsr should not be used either.
NOTE about $tsrSource param
In memoization scenarios where tokens are reused across source frames, we also need to reset the source objects to the target frame. Doing so effectively marks all SourceRange objects as belonging to the target frame. Note that the SourceRange design allows more fine-grained tracking across nested templates. Parsoid doesn't support that yet => the logic below is correct. But in a fine-grained tracking scenario, we'll need to either null offsets OR disable cross-frame memoization OR do more complicated state migration.
1.10.0