tools.itertools — Iterators for Efficient Looping#

Iterator functions.

Note

pairwise() function introduced in Python 3.10 is backported in backports

tools.itertools.filter_unique(iterable, container=None, key=None, add=None)[source]#

Yield unique items from an iterable, omitting duplicates.

By default, to provide uniqueness, it puts the generated items into a set created as a local variable. It only yields items which are not already present in the local set.

For large collections, this is not memory efficient, as a strong reference to every item is kept in a local set which cannot be cleared.

Also, the local set can’t be re-used when chaining unique operations on multiple generators.

To avoid these issues, it is advisable for the caller to provide their own container and set the key parameter to be the function hash, or use a weakref as the key.

The container can be any object that supports __contains__. If the container is a set or dict, the method add or __setitem__ will be used automatically. Any other method may be provided explicitly using the add parameter.

Beware that key=id is only useful for cases where id() is not unique.

Warning

This is not thread safe.

Added in version 3.0.

Parameters:

  • iterable (collections.abc.Iterable) – the source iterable

  • container (type) – storage of seen items

  • key (callable) – function to convert the item to a key

  • add (callable) – function to add an item to the container

tools.itertools.intersect_generators(*iterables, allow_duplicates=False)[source]#

Generator of intersect iterables.

Yield items only if they are yielded by all iterables. zip_longest is used to retrieve items from all iterables in parallel, so that items can be yielded before iterables are exhausted.

Generator is stopped when all iterables are exhausted. Quitting before all iterables are finished is attempted if there is no more chance of finding an item in all of them.

Sample:

>>> iterables = 'mississippi', 'missouri'
>>> list(intersect_generators(*iterables))
['m', 'i', 's']
>>> list(intersect_generators(*iterables, allow_duplicates=True))
['m', 'i', 's', 's', 'i']

Added in version 3.0.

Changed in version 5.0: Avoid duplicates (T263947).

Changed in version 6.4: genlist was renamed to iterables; consecutive iterables are to be used as iterables parameters or ‘*’ to unpack a list

Changed in version 7.0: Reimplemented without threads which is up to 10’000 times faster

Changed in version 9.0: Iterable elements may consist of lists or tuples allow_duplicates is a keyword-only argument

Parameters:

  • iterables – page generators

  • allow_duplicates (bool) – optional keyword argument to allow duplicates if present in all generators

tools.itertools.islice_with_ellipsis(iterable, *args, marker='…')[source]#

Generator which yields the first n elements of the iterable.

If more elements are available and marker is True, it returns an extra string marker as continuation mark.

Function takes the and the additional keyword marker.

Parameters:

  • iterable (iterable) – the iterable to work on

  • args – same args as: - itertools.islice(iterable, stop) - itertools.islice(iterable, start, stop[, step])

  • marker (str) – element to yield if iterable still contains elements after showing the required number. Default value: ‘…’

tools.itertools.itergroup(iterable, size, strict=False)[source]#

Make an iterator that returns lists of (up to) size items from iterable.

Example:

>>> i = itergroup(range(25), 10)
>>> print(next(i))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> print(next(i))
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> print(next(i))
[20, 21, 22, 23, 24]
>>> print(next(i))
Traceback (most recent call last):
 ...
StopIteration

Added in version 7.6: The strict parameter.

Deprecated since version 8.2: Use backports.batched() instead.

Parameters:

  • size (int) – How many items of the iterable to get in one chunk

  • strict (bool) – If True, raise a ValueError if length of iterable is not divisible by size.

Raises:

ValueError – iterable is not divisible by size

Return type:

Generator[list[Any], None, None]

tools.itertools.roundrobin_generators(*iterables)[source]#

Yield simultaneous from each iterable.

Sample:

>>> tuple(roundrobin_generators('ABC', range(5)))
('A', 0, 'B', 1, 'C', 2, 3, 4)

Added in version 3.0.

Changed in version 6.4: A sentinel variable is used to determine the end of an iterable instead of None.

Parameters:

iterables (iterable) – any iterable to combine in roundrobin way

Returns:

the combined generator of iterables

Return type:

generator