`textlib` — Changing Wikitext#

Functions for manipulating wiki-text.

class textlib.Content(header, sections, footer)[source]#

Bases: NamedTuple

A namedtuple as result of extract_sections() holding page content.

Changed in version 8.2: _Content becomes a public class.

Create new instance of Content(header, sections, footer)

Parameters:

header (str)
sections (list[Section])
footer (str)

footer: str#: the page footer

header: str#: the page header

sections: list[Section]#: the page sections

property title: str#

Return the first main title found on the page.

The first main title is anything enclosed within triple quotes.

Added in version 8.2.

class textlib.MultiTemplateMatchBuilder(site)[source]#

Bases: object

Build template matcher.

pattern(template, flags=re.DOTALL)[source]#: Return a compiled regex to match template.

search_any_predicate(templates)[source]#: Return a predicate that matches any template.

class textlib.Section(title, content)[source]#

Bases: NamedTuple

A namedtuple as part of Content describing a page section.

Changed in version 8.2: _Section becomes a public class.

Create new instance of Section(title, content)

Parameters:

title (str)
content (str)

content: str#: section content

property heading: str#: Return the section title without equal signs.

Added in version 8.2.

property level: int#: Return the section level.

Added in version 8.2.

title: str#: section title including equal signs

class textlib.TimeStripper(site=None)[source]#

Bases: object

Find timestamp in page and return it as pywikibot.Timestamp object.

Changed in version 8.0: group attribute is a set instead of a list. patterns is a TimeStripperPatterns namedtuple instead of a list.

Example:

>>> site = pywikibot.Site('wikipedia:fr')
>>> sign = 'Merci bien Xqt (d) 15 mai 2013 à 20:34 (CEST)'
>>> ts = TimeStripper(site)
>>> ts.timestripper(sign)  
Timestamp(2013, 5, 15, 20, 34, tzinfo=TZoneFixedOffset(3600, Europe/Paris))

static fix_digits(line)[source]#: Make non-latin digits like Persian to latin to parse.

Deprecated since version 7.0: Use to_latin_digits() instead.

property groups#: Deprecated groups attribute.

Deprecated since version 8.0: use textlib.TIMEGROUPS instead

property pdayR#: Deprecated day pattern attribute.

Deprecated since version 8.0: use patterns.day instead

property pmonthR#: Deprecated month pattern attribute.

Deprecated since version 8.0: use patterns.month instead

property ptimeR#: Deprecated time pattern attribute.

Deprecated since version 8.0: use pattern.time instead

property ptimeznR#: Deprecated tzinfo pattern attribute.

Deprecated since version 8.0: use patterns.tzinfo instead

property pyearR#: Deprecated year pattern attribute.

Deprecated since version 8.0: use patterns.year instead

timestripper(line)[source]#

Find timestamp in line and convert it to time zone aware datetime.

All the following items must be matched, otherwise None is returned: -. year, month, hour, time, day, minute, tzinfo

Changed in version 7.6: HTML parts are removed from line

Returns:: A timestamp found on the given line
Parameters:: line (str)
Return type:: Timestamp | None

class textlib.TimeStripperPatterns(time, tzinfo, year, month, day)[source]#

Bases: NamedTuple

Hold precompiled timestamp patterns for TimeStripper.

Attribute order is important to avoid mismatch when searching.

Added in version 8.0.

Create new instance of TimeStripperPatterns(time, tzinfo, year, month, day)

Parameters:

time (Pattern[str])
tzinfo (Pattern[str])
year (Pattern[str])
month (Pattern[str])
day (Pattern[str])

day: Pattern[str]#: Alias for field number 4

month: Pattern[str]#: Alias for field number 3

time: Pattern[str]#: Alias for field number 0

tzinfo: Pattern[str]#: Alias for field number 1

year: Pattern[str]#: Alias for field number 2

class textlib._GetDataHTML(*, convert_charrefs=True)[source]#

Bases: HTMLParser

HTML parser which removes html tags except they are listed in keeptags.

The parser is used by removeHTMLParts() similar to this:

from contextlib import closing
from pywikibot.textlib import _GetDataHTML
with closing(_GetDataHTML()) as parser:
    parser.keeptags = ['html']
    parser.feed('<html><head><title>Test</title></head>'
                '<body><h1><!-- Parse --> me!</h1></body></html>')
    print(parser.textdata)

The result is:

<html>Test me!</html>

Changed in version 9.2: This class is no longer a context manager; contextlib.closing() should be used instead.

See also

Initialize and reset this instance.

If convert_charrefs is True (the default), all character references are automatically converted to the corresponding Unicode characters.

handle_data(data)[source]#

Add data to text.

Return type:: None

handle_endtag(tag)[source]#

Add end tag to text if tag should be kept.

Return type:: None

handle_starttag(tag, attrs)[source]#

Add start tag to text if tag should be kept.

Return type:: None

keeptags: list[str] = []#

textdata = ''#

textlib._create_default_regexes()[source]#

Fill (and possibly overwrite) _regex_cache with default regexes.

The following keys are provided: category, comment, file, header, hyperlink, interwiki, invoke, link, pagelist, property, startcolon, startspace, table, template.

Return type:: None

textlib.add_text(text, add, *, site=None)[source]#

Add text to a page content above categories and interwiki.

Added in version 6.4.

Parameters:

text (str) – The page content to add text to.
add (str) – Text to add.
site (pywikibot.Site) – The site that the text is coming from. Required for reorder of categories and interlanguage links. Te default site is used otherwise.

Return type:

str

textlib.case_escape(case, string, *, underscore=False)[source]#

Return an escaped regex pattern which depends on ‘first-letter’ case.

Added in version 7.0.

Changed in version 8.4: Added the optional underscore parameter.

Parameters:

case (str) – if case is ‘first-letter’, the regex contains an inline re.IGNORECASE flag for the first letter
underscore (bool) – if True, expand the regex to detect spaces and underscores which are interchangeable and collapsible
string (str)

Return type:

str

textlib.categoryFormat(categories, insite=None)[source]#

Return a string containing links to all categories in a list.

Parameters:

categories (iterable) – A list of Category or Page objects or strings which can be either the raw name, [[Category:..]] or [[cat_localised_ns:…]].
insite (pywikibot.Site) – Used to to localise the category namespace.

Returns:

String of categories

Return type:

str

textlib.compileLinkR(withoutBracketed=False, onlyBracketed=False)[source]#

Return a regex that matches external links.

Parameters:

withoutBracketed (bool)
onlyBracketed (bool)

textlib.does_text_contain_section(pagetext, section)[source]#

Determine whether the page text contains the given section title.

It does not care whether a section string may contain spaces or underlines. Both will match.

If a section parameter contains an internal link, it will match the section with or without a preceding colon which is required for a text link e.g. for categories and files.

Parameters:

pagetext (str) – The wikitext of a page
section (str) – a section of a page including wikitext markups

Return type:

bool

textlib.expandmarker(text, marker='', separator='')[source]#

Return a marker expanded whitespace and the separator.

It searches for the first occurrence of the marker and gets the combination of the separator and whitespace directly before it.

Parameters:

text (str) – the text which will be searched.
marker (str) – the marker to be searched.
separator (str) – the separator string allowed before the marker. If empty it won’t include whitespace too.

Returns:

the marker with the separator and whitespace from the text in front of it. It’ll be just the marker if the separator is empty.

Return type:

str

textlib.extract_sections(text, site=None)[source]#

Return section headings and contents found in text.

The returned namedtuple Content contains the text parsed into header, sections and footer parts. The main title found in the header which is the first text enclosed with ‘’’ like ‘’’page title’’’ can be given by the title property.

The header part is a string containing text part above the first heading.

The sections part is a list of Section namedtuples, each tuple containing a string with section title (including equal signs), and a string with the section content. In addition the section heading (the title without equal signs) can be given by the heading property. Also the section level can be found by the level property which is the number of the equal signs around the section heading.

The footer part is also a string containing text part after the last section.

Examples:

>>> text = """
... '''this''' is a Python module.
...
... == History of this ==
... This set of principles was posted in 1999...
...
... == Usage of this ==
... Enter "import this" for usage...
...
... === Details ===
... The Zen of Python...
...
... [[Category:Programming principles]]
... """
>>> site = pywikibot.Site('wikipedia:en')
>>> result = extract_sections(text, site)
>>> result.header.strip()
"'''this''' is a Python module."
>>> result.sections[0].title
'== History of this =='
>>> result.sections[1].content.strip()
'Enter "import this" for usage...'
>>> result.sections[2].heading
'Details'
>>> result.sections[2].level
3
>>> result.footer.strip()
'[[Category:Programming principles]]'
>>> result.title
'this'

Note

sections and text from templates are not extracted but embedded as plain text.

Added in version 3.0.

Changed in version 8.2: The Content and Section class have additional properties.

Returns:

The parsed namedtuple.

Parameters:

text (str)
site (BaseSite | None)

Return type:

Content

textlib.extract_templates_and_params(text, remove_disabled_parts=False, strip=False)[source]#

Return a list of templates found in text.

Return value is a list of tuples. There is one tuple for each use of a template in the page, with the template title as the first entry and a dict of parameters as the second entry. Parameters are indexed by strings; as in MediaWiki, an unnamed parameter is given a parameter name with an integer value corresponding to its position among the unnamed parameters, and if this results multiple parameters with the same name only the last value provided will be returned.

This uses the package mwparserfromhell or wikitextparser as MediaWiki markup parser. mwparserfromhell is installed by default.

There are minor differences between the two implementations.

The parser packages preserves whitespace in parameter names and values.

If there are multiple numbered parameters in the wikitext for the same position, MediaWiki will only use the last parameter value. e.g. {{a| foo | 2  = bar | baz }} is {{a|1=foo|2=baz}} To replicate that behaviour, enable both remove_disabled_parts and strip parameters.

Parameters:

text (str) – The wikitext from which templates are extracted
remove_disabled_parts (bool) – If enabled, remove disabled wikitext such as comments and pre.
strip (bool) – If enabled, strip arguments and values of templates.

Returns:

list of template name and params

Return type:

list[tuple[str, OrderedDict[str, str]]]

Changed in version 6.1: wikitextparser package is supported; either wikitextparser or mwparserfromhell is strictly recommended.

textlib.extract_templates_and_params_regex_simple(text)[source]#

Extract top-level templates with params using only a simple regex.

This function uses only a single regex, and returns an entry for each template called at the top-level of the wikitext. Nested templates are included in the argument values of the top-level template.

This method will incorrectly split arguments when an argument value contains a ‘|’, such as {{template|a={{b|c}} }}.

Parameters:: text (str) – The wikitext from which templates are extracted
Returns:: list of template name and params
Return type:: list of tuple of name and OrderedDict

textlib.findmarker(text, startwith='@@', append=None)[source]#

Find a string which is not part of text.

Parameters:

text (str)
startwith (str)
append (str | None)

Return type:

str

textlib.getCategoryLinks(text, site=None, include=None, expand_text=False)[source]#

Return a list of category links found in text.

Parameters:

include (list[str] | None) – list of tags which should not be removed by removeDisabledParts() and where CategoryLinks can be searched.
text (str)
expand_text (bool)

Returns:

all category links found

Return type:

list[Category]

textlib.getLanguageLinks(text, insite=None, template_subpage=False)[source]#

Return a dict of inter-language links found in text.

The returned dict uses the site as keys and Page objects as values. It does not contain its own site.

Do not call this routine directly, use page.BasePage.interwiki() method instead.

Parameters:

text (str)
template_subpage (bool)

Return type:

dict[BaseSite, Page]

textlib.get_regexes(keys, site=None)[source]#

Fetch compiled regexes.

Changed in version 8.2: _get_regexes becomes a public function. keys may be a single string; site is optional.

Parameters:

keys (str | Iterable[str]) – a single key or an iterable of keys whose regex pattern should be given
site (BaseSite | None) – a BaseSite object needed for category, file, interwiki, invoke and property keys

Raises:

ValueError – site cannot be None.

Return type:

list[Pattern[str]]

textlib.glue_template_and_params(template_and_params)[source]#

Return wiki text of template glued from params.

You can use items from extract_templates_and_params here to get an equivalent template wiki text (it may happen that the order of the params changes).

Return type:: str

textlib.ignore_case(string)[source]#

Return a case-insensitive pattern for the string.

Changed in version 7.2: _ignore_case becomes a public method

Parameters:: string (str)
Return type:: str

textlib.interwikiFormat(links, insite=None)[source]#

Convert interwiki link dict into a wikitext string.

Parameters:

links (dict with the Site objects as keys, and Page or Link objects as values.) – interwiki links to be formatted
insite (BaseSite) – site the interwiki links will be formatted for (defaulting to the current site).

Returns:

string including wiki links formatted for inclusion in insite

Return type:

str

textlib.interwikiSort(sites, insite=None)[source]#: Sort sites according to local interwiki sort logic.

textlib.isDisabled(text, index, tags=None)[source]#

Return True if text[index] is disabled, e.g. by a comment or nowiki tag.

For the tags parameter, see removeDisabledParts.

Parameters:

text (str)
index (int)

Return type:

bool

textlib.reformat_ISBNs(text, match_func)[source]#

Reformat ISBNs.

Parameters:

text (str) – text containing ISBNs
match_func (callable) – function to reformat matched ISBNs

Returns:

reformatted text

Return type:

str

textlib.removeCategoryLinks(text, site=None, marker='')[source]#

Return text with all category links removed.

Parameters:

text (str) – The text that needs to be modified.
site (pywikibot.Site) – The site that the text is coming from.
marker (str) – If defined, marker is placed after the last category link, or at the end of text if there are no category links.

Returns:

The modified text.

Return type:

str

textlib.removeCategoryLinksAndSeparator(text, site=None, marker='', separator='')[source]#

Return text with category links and preceding separators removed.

Parameters:

text (str) – The text that needs to be modified.
site (pywikibot.Site) – The site that the text is coming from.
marker (str) – If defined, marker is placed after the last category link, or at the end of text if there are no category links.
separator (str) – The separator string that will be removed if followed by the category links.

Returns:

The modified text

Return type:

str

textlib.removeDisabledParts(text, tags=None, include=None, site=None)[source]#

Return text without portions where wiki markup is disabled.

Parts that will be removed by default are:

HTML comments
nowiki tags
pre tags
includeonly tags
source and syntaxhighlight tags

Changed in version 7.0: the order of removals will correspond to the tags argument if provided as an ordered collection (list, tuple)

Parameters:

tags (Iterable | None) – The exact set of parts which should be removed using keywords from get_regexes().
include (Container | None) – Or, in alternative, default parts that shall not be removed.
site (BaseSite | None) – Site to be used for site-dependent regexes. Default disabled parts listed above do not need it.
text (str)

Returns:

text stripped from disabled parts.

Return type:

str

textlib.removeHTMLParts(text, keeptags=None)[source]#

Return text without portions where HTML markup is disabled.

Parts that can/will be removed are HTML tags and all wiki tags. The exact set of parts which should NOT be removed can be passed as the keeptags parameter, which defaults to ['tt', 'nowiki', 'small', 'sup'].

Example:

>>> removeHTMLParts('<div><b><ref><tt>Hi all!</tt></ref></b></div>')
'<tt>Hi all!</tt>'

textlib — Changing Wikitext#

`textlib` — Changing Wikitext#