pywikibot.comms package

Communication layer.

Submodules

pywikibot.comms.eventstreams module

Server-Sent Events client.

This file is part of the Pywikibot framework.

This module requires sseclient to be installed:

pip install sseclient
class pywikibot.comms.eventstreams.EventStreams(**kwargs, stream='[deprecated name of streams]')[source]

Bases: object

Basic EventStreams iterator class for Server-Sent Events (SSE) protocol.

It provides access to arbitrary streams of data including recent changes. It replaces rcstream.py implementation.

Usage:

>>> stream = EventStreams(streams='recentchange')
>>> stream.register_filter(type='edit', wiki='wikidatawiki')
>>> change = next(iter(stream))
>>> print('{type} on page {title} by {user}.'.format(**change))
edit on page Q32857263 by XXN-bot.
>>> change
{'comment': '/* wbcreateclaim-create:1| */ [[Property:P31]]: [[Q4167836]]',
 'wiki': 'wikidatawiki', 'type': 'edit', 'server_name': 'www.wikidata.org',
 'server_script_path': '/w', 'namespace': 0, 'title': 'Q32857263',
 'bot': True, 'server_url': 'https://www.wikidata.org',
 'length': {'new': 1223, 'old': 793},
 'meta': {'domain': 'www.wikidata.org', 'partition': 0,
          'uri': 'https://www.wikidata.org/wiki/Q32857263',
          'offset': 288986585, 'topic': 'eqiad.mediawiki.recentchange',
          'request_id': '1305a006-8204-4f51-a27b-0f2df58289f4',
          'schema_uri': 'mediawiki/recentchange/1',
          'dt': '2017-07-13T10:55:31+00:00',
          'id': 'ca13742b-67b9-11e7-935d-141877614a33'},
 'user': 'XXN-bot', 'timestamp': 1499943331, 'patrolled': True,
 'id': 551158959, 'minor': False,
 'revision': {'new': 518751558, 'old': 517180066}}
>>> del stream
__init__(**kwargs, stream='[deprecated name of streams]')[source]

Initializer.

Keyword Arguments:
 
  • site – a project site object. Used when no url is given
  • since – a timestamp for older events; there will likely be between 7 and 31 days of history available but is not guaranteed. It may be given as a pywikibot.Timestamp, an ISO 8601 string or a mediawiki timestamp string.
  • streams – event stream types. Mandatory when no url is given. Multiple streams may be given as a string with comma separated stream types or an iterable of strings Refer https://stream.wikimedia.org/?doc for available wikimedia stream types.
  • timeout – a timeout value indication how long to wait to send data before giving up
  • url – an url retrieving events from. Will be set up to a default url using _site.family settings, stream types and timestamp
Parameters:

kwargs – keyword arguments passed to SSEClient and requests lib

Raises:
  • ImportError – sseclient is not installed
  • NotImplementedError – no stream types specified
__iter__()[source]

Iterator.

__module__ = 'pywikibot.comms.eventstreams'
__repr__()[source]

Return representation string.

register_filter(*args, **kwargs)[source]

Register a filter.

Filter types:

There are 3 types of filter: ‘all’, ‘any’ and ‘none’. The filter type must be given with the keyword argument ‘ftype’ (see below). If no ‘ftype’ keyword argument is given, ‘all’ is assumed as default.

You may register multiple filters for each type of filter. The behaviour of filter type is as follows:

- B{'none'}: Skip if the any filter matches. Otherwise check 'all'.
- B{'all'}: Skip if not all filter matches. Otherwise check 'any':
- B{'any'}: Skip if no given filter matches. Otherwise pass.

Filter functions:

Filter may be specified as external function methods given as positional argument like:

def foo(data):
    return True

register_filter(foo, ftype='any')

The data dict from event is passed to the external filter function as a parameter and that method must handle it in a proper way and return True if the filter matches and False otherwise.

Filter keys and values:

Another method to register a filter is to pass pairs of keys and values as keyword arguments to this method. The key must be a key of the event data dict and the value must be any value or an iterable of values the data['key'] may match or be part of it. Samples:

register_filter(server_name='de.wikipedia.org')  # 1
register_filter(type=('edit', 'log'))  # 2
register_filter(ftype='none', bot=True)  # 3

Explanation for the result of the filter function: 1. return data['sever_name'] == 'de.wikipedia.org' 2. return data['type'] in ('edit', 'log') 3. return data['bot'] is True

Keyword Arguments:
 

ftype – The filter type, one of ‘all’, ‘any’, ‘none’. Default value is ‘all’

Parameters:
  • args (callable) – You may pass your own filter functions here. Every function should be able to handle the data dict from events.
  • kwargs (str, list, tuple or other sequence) – Any key returned by event data with a event data value for this given key.
Raises:

TypeError – A given args parameter is not a callable.

set_maximum_items(value)[source]

Set the maximum number of items to be retrieved from the stream.

If not called, most queries will continue as long as there is more data to be retrieved from the stream.

Parameters:value (int) – The value of maximum number of items to be retrieved in total to set.
streamfilter(data)[source]

Filter function for eventstreams.

See the description of register_filter() how it works.

Parameters:data (dict) – event data dict used by filter functions
url

Get the EventStream’s url.

Raises:NotImplementedError – no stream types specified
pywikibot.comms.eventstreams.site_rc_listener(site, total=None)[source]

Yield changes received from EventStream.

Parameters:
  • site (Pywikibot.BaseSite) – the Pywikibot.Site object to yield live recent changes for
  • total (int) – the maximum number of changes to return
Returns:

pywikibot.comms.eventstream.rc_listener configured for given site

Raises:

ImportError – sseclient installation is required

pywikibot.comms.http module

Basic HTTP access interface.

This module handles communication between the bot and the HTTP threads.

This module is responsible for
  • Setting up a connection pool
  • Providing a (blocking) interface for HTTP requests
  • Translate site objects with query strings into URLs
  • URL-encoding all data
  • Basic HTTP error handling
pywikibot.comms.http.error_handling_callback(request)[source]

Raise exceptions and log alerts.

Parameters:request (threadedhttp.HttpRequest) – Request that has completed
pywikibot.comms.http.fake_user_agent()[source]

Return a fake user agent.

Return type:str
pywikibot.comms.http.fetch(uri, method='GET', params=None, body=None, headers=None, default_error_handling=True, use_fake_user_agent=False, data=None, **kwargs)[source]

Blocking HTTP request.

Note: The callback runs in the HTTP thread, where exceptions are logged but are not able to be caught.

See requests.Session.request for parameters.

Parameters:
  • default_error_handling (bool) – Use default error handling
  • use_fake_user_agent (bool, str) – Set to True to use fake UA, False to use pywikibot’s UA, str to specify own UA. This behaviour might be overridden by domain in config.
Return type:

threadedhttp.HttpRequest

pywikibot.comms.http.get_authentication(uri)[source]

Retrieve authentication token.

Parameters:uri (str) – the URI to access
Returns:authentication token
Return type:None or tuple of two str
pywikibot.comms.http.get_fake_user_agent()[source]

Return a fake user agent depending on fake_user_agent option in config.

Deprecated, use fake_user_agent() instead.

Return type:str
pywikibot.comms.http.mode_check_decorator(func)[source]

Decorate load()/save() CookieJar methods.

pywikibot.comms.http.request(site=None, uri=None, method='GET', params=None, body=None, headers=None, data=None, **kwargs, ssl=NotImplemented)[source]

Request to Site with default error handling and response decoding.

See requests.Session.request for additional parameters.

If the site argument is provided, the uri is a relative uri from and including the document root ‘/’.

If the site argument is None, the uri must be absolute.

Parameters:
Keyword Arguments:
 

charset – Either a valid charset (usable for str.decode()) or None to automatically chose the charset from the returned header (defaults to latin-1)

Returns:

The received data

Return type:

a unicode string

pywikibot.comms.http.user_agent(site=None, format_string=None)[source]

Generate the user agent string for a given site and format.

Parameters:
  • site (BaseSite) – The site for which this user agent is intended. May be None.
  • format_string (basestring) – The string to which the values will be added using str.format. Is using config.user_agent_format when it is None.
Returns:

The formatted user agent

Return type:

str

pywikibot.comms.http.user_agent_username(username=None)[source]

Reduce username to a representation permitted in HTTP headers.

To achieve that, this function: 1) replaces spaces (‘ ‘) with ‘_’ 2) encodes the username as ‘utf-8’ and if the username is not ASCII 3) URL encodes the username if it is not ASCII, or contains ‘%’

pywikibot.comms.threadedhttp module

Http backend layer, formerly providing a httplib2 wrapper.

class pywikibot.comms.threadedhttp.HttpRequest(uri, method='GET', params=None, body=None, headers=None, callbacks=None, charset=None, **kwargs)[source]

Bases: pywikibot.tools.UnicodeMixin

Object wrapper for HTTP requests that need to block origin thread.

self.data will be either: * a tuple of (dict, unicode) if the request was successful * an exception

__bytes__()[source]

Return the undecoded response.

__init__(uri, method='GET', params=None, body=None, headers=None, callbacks=None, charset=None, **kwargs)[source]

Initializer.

See Http.request for parameters.

__module__ = 'pywikibot.comms.threadedhttp'
__unicode__()[source]

Return the response decoded by the detected encoding.

data

Return the requests response tuple.

decode(encoding, errors='strict')[source]

Return the decoded response.

encoding

Detect the response encoding.

exception

Get the exception, if any.

header_encoding

Return charset given by the response header.

hostname

Return the host of the request.

parsed_uri

Return the parsed requested uri.

raw

Return the raw response body.

response_headers

Return the response headers.

status

Return the HTTP response status.

Return type:int
text

Return the response decoded by the detected encoding.