comms — Communication layer

Communication layer.

comms.eventstreams — Server-Sent Events Client

Server-Sent Events client.

This file is part of the Pywikibot framework.

This module requires sseclient to be installed:

pip install "sseclient<0.0.23,>=0.0.18"

Added in version 3.0.

class comms.eventstreams.EventStreams(**kwargs)

Bases: GeneratorWrapper

Generator class for Server-Sent Events (SSE) protocol.

It provides access to arbitrary streams of data including recent changes.

Usage:

>>> stream = EventStreams(streams='recentchange')
>>> stream.register_filter(type='edit', wiki='wikidatawiki', bot=True)
>>> change = next(stream)
>>> msg = '{type} on page {title} by {user}.'.format_map(change)
>>> print(msg)  
edit on page Q2190037 by KrBot.
>>> from pprint import pprint
>>> pprint(change, width=75)  
{'$schema': '/mediawiki/recentchange/1.0.0',
 'bot': True,
 'comment': '/* wbsetreference-set:2| */ [[Property:P10585]]: 96FPN, см. '
            '/ see [[Template:Autofix|autofix]] на / on [[Property '
            'talk:P356]]',
 'id': 1728475074,
 'length': {'new': 8871, 'old': 8871},
 'meta': {'domain': 'www.wikidata.org',
          'dt': '2022-07-12T17:54:15Z',
          'id': '2cdec62f-a2b3-49b8-9a52-85a42236fb99',
          'offset': 4000957901,
          'partition': 0,
          'request_id': 'f7896e77-fd2b-4a95-a9e4-44c1e3ad818b',
          'stream': 'mediawiki.recentchange',
          'topic': 'eqiad.mediawiki.recentchange',
          'uri': 'https://www.wikidata.org/wiki/Q2190037'},
 'minor': False,
 'namespace': 0,
 'parsedcomment': '‎<span dir="auto"><span '
                  'class="autocomment">Изменена ссылка на заявление: '
                  '</span></span> <a href="/wiki/Property:P10585" '
                  'title="Property:P10585">Property:P10585</a>: 96FPN, '
                  'см. / see <a href="/wiki/Template:Autofix" '
                  'title="Template:Autofix">autofix</a> на / on <a '
                  'href="/wiki/Property_talk:P356" title="Property '
                  'talk:P356">Property talk:P356</a>',
 'patrolled': True,
 'revision': {'new': 1676015019, 'old': 1675697125},
 'server_name': 'www.wikidata.org',
 'server_script_path': '/w',
 'server_url': 'https://www.wikidata.org',
 'timestamp': 1657648455,
 'title': 'Q2190037',
 'type': 'edit',
 'user': 'KrBot',
 'wiki': 'wikidatawiki'}
>>> pprint(next(stream), width=75)  
{'$schema': '/mediawiki/recentchange/1.0.0',
 'bot': True,
 ...
 'server_name': 'www.wikidata.org',
 'server_script_path': '/w',
 'server_url': 'https://www.wikidata.org',
 ...
 'type': 'edit',
 'user': ...,
 'wiki': 'wikidatawiki'}
>>> del stream

Changed in version 7.6: subclassed from tools.collections.GeneratorWrapper

Keyword Arguments:

• canary (bool) – if True, include canary events, see https://w.wiki/7$2z for more info

• site (APISite) – a project site object. Used if no url is given

• since (Timestamp or str) – a timestamp for older events; there will likely be between 7 and 31 days of history available but is not guaranteed. It may be given as a pywikibot.Timestamp, an ISO 8601 string or a mediawiki timestamp string.

• streams (Iterable[str] or str) – event stream types. Mandatory when no url is given. Multiple streams may be given as a string with comma separated stream types or an iterable of strings

• timeout (int or float or tuple[int or float, int or float]) – a timeout value indication how long to wait to send data before giving up

• url (str) – an url retrieving events from. Will be set up to a default url using _site.family settings, stream types and timestamp

Parameters:

kwargs – keyword arguments passed to SSEClient and requests library

Raises:

• ImportError – sseclient is not installed

• NotImplementedError – no stream types specified

See also

https://stream.wikimedia.org/?doc#streams for available Wikimedia stream types to be passed with streams parameter.

property generator

Inner generator.

Changed in version 7.6: changed from iterator method to generator property

register_filter(*args, **kwargs)

Register a filter.

Filter types:

There are 3 types of filter: ‘all’, ‘any’ and ‘none’. The filter type must be given with the keyword argument ‘ftype’ (see below). If no ‘ftype’ keyword argument is given, ‘all’ is assumed as default.

You may register multiple filters for each type of filter. The behaviour of filter type is as follows:

• ‘none’: Skip if the any filter matches. Otherwise check ‘all’.

• ‘all’: Skip if not all filter matches. Otherwise check ‘any’:

• ‘any’: Skip if no given filter matches. Otherwise pass.

Filter functions:

Filter may be specified as external function methods given as positional argument like:

def foo(data):
    return True

register_filter(foo, ftype='any')

The data dict from event is passed to the external filter function as a parameter and that method must handle it in a proper way and return True if the filter matches and False otherwise.

Filter keys and values:

Another method to register a filter is to pass pairs of keys and values as keyword arguments to this method. The key must be a key of the event data dict and the value must be any value or an iterable of values the data['key'] may match or be part of it. Samples:

register_filter(server_name='de.wikipedia.org')  # 1
register_filter(type=('edit', 'log'))  # 2
register_filter(ftype='none', bot=True)  # 3

Explanation for the result of the filter function: 1. return data['sever_name'] == 'de.wikipedia.org' 2. return data['type'] in ('edit', 'log') 3. return data['bot'] is True

Keyword Arguments:

ftype – The filter type, one of ‘all’, ‘any’, ‘none’. Default value is ‘all’

Parameters:

• args (callable) – You may pass your own filter functions here. Every function should be able to handle the data dict from events.

• kwargs (str, list, tuple or other sequence) – Any key returned by event data with an event data value for this given key.

Raises:

TypeError – A given args parameter is not a callable.

set_maximum_items(value)

Set the maximum number of items to be retrieved from the stream.

If not called, most queries will continue as long as there is more data to be retrieved from the stream.

Parameters:

value (int) – The value of maximum number of items to be retrieved in total to set.

Return type:

None

streamfilter(data)

Filter function for eventstreams.

See the description of register_filter() how it works.

Parameters:

data (dict) – event data dict used by filter functions

property url

Get the EventStream’s url.

Raises:

NotImplementedError – no stream types specified

comms.eventstreams.site_rc_listener(site, total=None)

Yield changes received from EventStream.

Parameters:

• site (Pywikibot.BaseSite) – the Pywikibot.Site object to yield live recent changes for

• total (int | None) – the maximum number of changes to return

Returns:

pywikibot.comms.eventstream.rc_listener configured for given site

Raises:

ImportError – sseclient installation is required

comms.http — HTTP access interface

Basic HTTP access interface.

This module handles communication between the bot and the HTTP threads.

This module is responsible for

• Setting up a connection pool

• Providing a (blocking) interface for HTTP requests

• Translate site objects with query strings into URLs

• URL-encoding all data

• Basic HTTP error handling

This module creates and uses its own requests.Session object. The session is closed if the module terminates. If required you can use your own Session object passing it to the http.session variable:

from pywikibot.comms import http
session = requests.Session()
http.session = session

To enable access via cookies, assign cookie handling class:

session.cookies = http.cookie_jar

Changed in version 8.0: Cookies are lazy loaded when logging to site.

class comms.http.PywikibotCookieJar(filename=None, delayload=False, policy=None)

Bases: LWPCookieJar

CookieJar which create the filename and checks file permissions.

Added in version 8.0.

Cookies are NOT loaded from the named file until either the .load() or .revert() method is called.

load(user='', *args, **kwargs)

Loads cookies from a file.

Insert the account name to the cookie filename, set the instance`s filename and load the cookies.

Parameters:

user (str) – account name to be part of the cookie filename.

Return type:

None

save(*args, **kwargs)

Check the file mode and save cookies to a file.

Note

PywikibotCookieJar must be loaded previously to set the filename.

Raises:

ValueError – a filename was not supplied; load() must be called first.

Return type:

None

comms.http.cookie_jar = <PywikibotCookieJar[]>

global PywikibotCookieJar instance.

comms.http.error_handling_callback(response)

Raise exceptions and log alerts.

Parameters:

response (requests.Response) – Response returned by Session.request().

comms.http.fake_user_agent()

Return a fake user agent.

Return type:

str

comms.http.fetch(uri, method='GET', headers=None, default_error_handling=True, use_fake_user_agent=False, **kwargs)

HTTP request.

See requests.Session.request for parameters.

Parameters:

• uri (str) – URL to send

• method (str) – HTTP method of the request (default: GET)

• headers (dict | None) – dictionary of headers of the request

• default_error_handling (bool) – Use default error handling

• use_fake_user_agent (bool | str) – Set to True to use fake UA, False to use pywikibot’s UA, str to specify own UA. This behaviour might be overridden by domain in config.

Keyword Arguments:

• charset – Either a valid charset (usable for str.decode()) or None to automatically chose the charset from the returned header (defaults to latin-1)

• verify – verify the SSL certificate (default is True)

• callbacks – Methods to call once data is fetched

Return type:

requests.Response

comms.http.flush()

Close the session object. This is called when the module terminates.

Changed in version 8.1: log the traceback and show the exception value in the critical message

Return type:

None

comms.http.get_authentication(uri)

Retrieve authentication token.

Parameters:

uri (str) – the URI to access

Returns:

authentication token

Return type:

tuple[str, str] | None

comms.http.get_charset_from_content_type(content_type)

Get charset from the content-type header.

Added in version 7.3.

Parameters:

content_type (str)

Return type:

str | None

comms.http.request(site, uri=None, headers=None, **kwargs)

Request to Site with default error handling and response decoding.

See requests.Session.request for additional parameters.

The optional uri is a relative uri from site base uri including the document root ‘/’.

Changed in version 8.2: a protocol parameter can be given which is passed to the family.Family.base_url() method.

Parameters:

• site (BaseSite) – The Site to connect to

• uri (str | None) – the URI to retrieve

• headers (dict | None)

Keyword Arguments:

• charset (Optional[CodecInfo, str]) – Either a valid charset (usable for str.decode()) or None to automatically chose the charset from the returned header (defaults to latin-1)

• protocol (Optional[str]) – a url scheme

Returns:

The received data Response

Return type:

Response

comms.http.session = <requests.sessions.Session object>

global requests.Session.

comms.http.user_agent(site=None, format_string='')

Generate the user agent string for a given site and format.

Parameters:

• site (BaseSite | None) – The site for which this user agent is intended. May be None.

• format_string (str) – The string to which the values will be added using str.format. Is using config.user_agent_format when it is empty.

Returns:

The formatted user agent

Return type:

str

comms.http.user_agent_username(username=None)

Reduce username to a representation permitted in HTTP headers.

To achieve that, this function: 1) replaces spaces (’ ‘) with ‘_’ 2) encodes the username as ‘utf-8’ and if the username is not ASCII 3) URL encodes the username if it is not ASCII, or contains ‘%’