PuppetDB¶

PuppetDB backend.

cumin.backends.puppetdb.CATEGORIES = ('C', 'F', 'O', 'P', 'R')¶

available categories in the grammar.

C: shortcut for querying resources of type Class, equivalent of R:Class = class_path`.
F: for querying facts.
O: shortcut for querying resources of type Class that starts with Role::.
P: shortcut for querying resources of type Class that starts with Profile::.
R: for querying generic resources.

Type:: tuple()

cumin.backends.puppetdb.OPERATORS = ('=', '>=', '<=', '<', '>', '~')¶

available operators in the grammar, the same available in PuppetDB API.

The ~ one is used for regex matching.

Type:: tuple()

class cumin.backends.puppetdb.ParsedString(string, is_quoted)[source]¶

Bases: object

Simple string wrapper which can communicate if a string should be enquoted downstream.

Constructor for ParsedString.

Parameters:

string (str) -- The string to store in this object.
is_quoted (bool) -- Whether the output should be quoted when this is converted to a string.

capwords(sep)[source]¶

Perform capwords operation on internal value and return a new ParsedString.

Parameters:: according to string.capwords().

replace(old, new, count=-1)[source]¶

Perform replace operation on internal value and return a new ParsedString.

Parameters:: according to str.replace().

cumin.backends.puppetdb.grammar()[source]¶

Define the query grammar.

Backus-Naur form (BNF) of the grammar:

    <grammar> ::= <item> | <item> <and_or> <grammar>
       <item> ::= [<neg>] <query-token> | [<neg>] "(" <grammar> ")"
<query-token> ::= <token> | <hosts>
      <token> ::= <category>:<key> [<operator> <value>]
      <value> ::= <numeric> | <bareword> | <quoted_string> | <unquoted_string>

Given that the pyparsing library defines the grammar in a BNF-like style, for the details of the tokens not specified above check directly the source code.

Returns:: the grammar parser.
Return type:: pyparsing.ParserElement

class cumin.backends.puppetdb.PuppetDBQuery(config)[source]¶

Bases: BaseQuery

PuppetDB query builder.

The puppetdb backend allow to use an existing PuppetDB instance for the hosts selection. The supported PuppetDB API version is 4.

Each query part can be composed with the others using boolean operators (and, or, not)
Multiple query parts can be grouped together with parentheses ((, )).
A query part can be of two different types:
- Hostname matching: this is a simple string that be used to match directly the hostname of the hosts in the selected backend. It allows for glob expansion (*) and the use of the powerful ClusterShell.NodeSet.NodeSet.
- Category matching: an identifier composed by a category, a colon and a key, followed by a comparison operator and a value, as in F:key = value.
Values may be of various types supported by PuppetDB (numerics, boolean, and strings) for example:
- Booleans: true, false
- Strings: 'a string', and unquoted single words that aren't true or false and do not start with an integer.
- Numeric values: 15, 23.5, 0, 0xfa Note: hexadecimal and octal numbers are supported by cumin but converted into normal integers. Some fields in PuppetDB may have hex or octal stored as strings, and should be quoted such as '0xfa'.
Note: PuppetDB may or may not support a particular value type for a particular resource.

Some query examples:

All hosts: *
Hosts globbing: host10*
ClusterShell.NodeSet.NodeSet syntax for hosts expansion: host10[10-42].domain
Category based key-value selection:
- R:Resource::Name: query all the hosts that have a resource of type Resource::Name.
- R:Resource::Name = 'resource-title': query all the hosts that have a resource of type Resource::Name whose title is resource-title. For example R:Class = MyModule::MyClass.
- R:Resource::Name@field = 'some-value': query all the hosts that have a resource of type Resource::Name whose field field has the value some-value. The valid fields are: tag, certname, type, title, exported, file, line. The previous syntax is a shortcut for this one with the field title.
- R:Resource::Name%param = 'some-value': query all the hosts that have a resource of type Resource::Name whose parameter param has the value some-value.
- C:Class::Name: special shortcut to query all the hosts that have a resource of type Class whose name is Class::Name. The Class::Name part is completely arbitrary and depends on the puppet hierarchy chosen. It's equivalent to R:Class = Class::Name, with the addition that the param and field selectors described above can be used directly without the need to add another condition.
- O:Module::Name: special shortcut to query all the hosts that have a resource of type Class whose name is Role::Module::Name. The Module::Name part is completely arbitrary and depends on the puppet hierarchy chosen. It's equivalent to R:Class = Role::Module::Name, with the addition that the param and field selectors described above can be used directly without the need to add another condition, although usually roles should not have parameters in the role/profile Puppet paradigm.
- P:Module::Name: special shortcut to query all the hosts that have a resource of type Class whose name is Profile::Module::Name. The Module::Name part is completely arbitrary and depends on the puppet hierarchy chosen. It's equivalent to R:Class = Profile::Module::Name, with the addition that the param and field selectors described above can be used directly without the need to add another condition.
- F:FactName = value: query all the hosts that have a fact FactName, as reported by facter, with the value value.
- Mixed facts/resources queries are not supported, but the same result can be achieved using the main grammar with multiple subqueries for the PuppetDB backend.
All hosts with physicalcorecount fact greater than 2: F:physicalcorecount > 2
A complex selection for facts: host10[10-42].*.domain or (not F:key1 = value1 and host10*) or (F:key2 > value2 and F:key3 ~ '^value[0-9]+')

Query constructor for the PuppetDB backend.

Parameters:: according to parent cumin.backends.BaseQuery.__init__().

base_url_template = '{scheme}://{host}:{port}/pdb/query/v4/'¶

string template in the str.format() style used to generate the base URL of the PuppetDB server.

Type:: str

endpoints = {'C': 'resources', 'F': 'nodes', 'O': 'resources', 'P': 'resources', 'R': 'resources'}¶

dictionary with the mapping of the available categories in the grammar to the PuppetDB API endpoints.

Type:: dict

category_prefixes = {'C': '', 'O': 'Role', 'P': 'Profile'}¶

dictionary with the mapping of special categories to title prefixes.

Type:: dict

grammar = Forward: {{Group:({['not'] {{Combine:({{C | F | O | P | R ':'} W:(%-.0-:@-Z_a-z)}) [= | >= | <= | < | > | ~ {{{{Re:('0x[0-9A-F]+') ^ W:(0, 0-7){2,...}} ^ number} ^ true | false} ^ {quoted string using single or double quotes ^ W:(!-'*-z|~)}}]} | {quoted string using single or double quotes | {~{{{'and' | 'or'} | 'not'}} W:(!&*,-.0-9A-[]...)}}}}) | Group:({{{['not'] '('} : ...} ')'})} [{Group:({'and' | 'or'}) : ...}]...}¶

load the grammar parser only once in a singleton-like way.

Type:: pyparsing.ParserElement

property endpoint¶

Endpoint in the PuppetDB API for the current query.

Getter:: Returns the current endpoint or a default value if not set.
Setter:: str: the value to set the endpoint to.
Raises:: cumin.backends.InvalidQueryError -- if trying to set it to an invalid endpoint or mixing endpoints in a single query.

_open_subgroup()[source]¶: Handle subgroup opening.

_close_subgroup()[source]¶: Handle subgroup closing.

static _get_grouped_tokens()[source]¶

Return an empty grouped tokens structure.

Returns:: the dictionary with the empty grouped tokens structure.
Return type:: dict

_build(query_string)[source]¶

Override parent class _build method to reset tokens and add logging.

Parameters:: according to parent cumin.backends.BaseQuery._build().

_execute()[source]¶

Concrete implementation of parent abstract method.

Parameters:: according to parent cumin.backends.BaseQuery._execute().
Returns:: with the FQDNs of the matching hosts.
Return type:: ClusterShell.NodeSet.NodeSet

_add_category(*, category, key, value=None, operator='=', neg=False)[source]¶

Add a category token to the query 'F:key = value'.

Parameters:

category (str) -- the category of the token, one of CATEGORIES.
key (str) -- the key for this category.
value (str, optional) -- the value to match, if not specified the key itself will be matched.
operator (str, optional) -- the comparison operator to use, one of OPERATORS.
neg (bool, optional) -- whether the token must be negated.

Raises:

cumin.backends.InvalidQueryError -- on internal parsing error.

_add_hosts(hosts, neg=False)[source]¶

Add a list of hosts to the query.

Parameters:

hosts (list) -- list of ClusterShell.NodeSet.NodeSet with the list of hosts to search.
neg (bool, optional) -- whether the token must be negated.

_parse_token(token)[source]¶

Concrete implementation of parent abstract method.

Parameters:: according to parent cumin.backends.BaseQuery._parse_token().
Raises:: cumin.backends.InvalidQueryError -- on internal parsing error.

_get_resource_query(key, value=None, operator='=')[source]¶

Build a resource query based on the parameters, resolving the special cases for %params and @field.

Parameters:

key (str) -- the key of the resource.
value (str, optional) -- the value to match, if not specified the key itself will be matched.
operator (str, optional) -- the comparison operator to use, one of OPERATORS.

Returns:

the resource query.

Return type:

str

Raises:

cumin.backends.InvalidQueryError -- on invalid combinations of parameters.

_get_special_resource_query(category, key, value, operator)[source]¶

Build a query for Roles and Profiles, resolving the special cases for %params and @field.

Parameters:

category (str) -- the category of the token, one of category_prefixes keys.
key (str) -- the key of the resource to use as a suffix for the Class title matching.
value (str, optional) -- the value to match in case %params or @field is specified.
operator (str, optional) -- the comparison operator to use if there is a value, one of OPERATORS.

Returns:

the resource query.

Return type:

str

Raises:

cumin.backends.InvalidQueryError -- on invalid combinations of parameters.

_get_query_string(group)[source]¶

Recursively build and return the PuppetDB query string.

Parameters:: group (dict) -- a dictionary with the grouped tokens.
Returns:: the query string for the PuppetDB API.
Return type:: str

_add_bool(bool_op)[source]¶

Add a boolean AND or OR query block to the query and validate logic.

Parameters:: bool_op (str) -- the boolean operator to add to the query: and, or.
Raises:: cumin.backends.InvalidQueryError -- if an invalid boolean operator was found.

_api_call(query)[source]¶

Execute a query to PuppetDB API and return the parsed JSON.

Parameters:: query (str) -- the query parameter to send to the PuppetDB API.
Raises:: requests.HTTPError -- if the PuppetDB API call fails.

cumin.backends.puppetdb.GRAMMAR_PREFIX = 'P'¶

the prefix associate to this grammar, to register this backend into the general grammar. Required by the backend auto-loader in cumin.grammar.get_registered_backends().

Type:: str

cumin.backends.puppetdb.query_class¶: Required by the backend auto-loader in cumin.grammar.get_registered_backends().