Defined Type: burrow

Defined in:
modules/burrow/manifests/init.pp

Overview

SPDX-License-Identifier: Apache-2.0

Define: burrow

Burrow is a consumer offset lag monitoring tool for Kafka This module helps set up a burrow service that provides a http endpoint to query, and will email notifications on consumer groups statuses. This module supports only Burrow >= 1.0, since older versions are based on completely different configurations.

Parameters

client_id

The client ID string to provide to Kafka when consuming

httpserver_port

Port at which to make the burrow http endpoint available

lagcheck_intervals

Length of window of offsets used to monitor lag See: github.com/linkedin/Burrow/wiki/Configuration#lagcheck

zookeeper_hosts

Array of zookeeper host and their ports.

zookeeper_path

The full path to the znode that is the root for the Kafka cluster.

kafka_cluster_name

Name of the Kafka cluster to monitor.

kafka_brokers

Array of kafka brokers in the Kafka cluster.

kafka_api_version

Kafka api version to use with the cluster. Current maximum supported one is 1.0.0 Default: '1.0.0'

alert_whitelist

Regex related to a whitelist of consumer groups that can trigger notifications via email.

smtp_server

SMTP server to send emails from

from_email

From email address for notification

to_email

Email address to send email notification to

email_template

The name of the email template to use for Burrow's alerts

consumer_groups_blacklist

Regex used to filter out temporary/not-relevant consumer groups. Default: '^(console-consumer-|python-kafka-consumer-|test_).*$'

kafka_brokers_port

Port used by Kafka brokers. Default: 9092

zookeeper_port

Port used by zookeeper. Default: 2181

Parameters:

  • zookeeper_hosts (Any)
  • zookeeper_path (Any)
  • kafka_cluster_name (Any)
  • kafka_brokers (Any)
  • alert_whitelist (Any)
  • smtp_server (Any)
  • from_email (Any)
  • to_email (Any)
  • smtp_server_port (Any) (defaults to: 25)
  • kafka_brokers_port (Any) (defaults to: 9092)
  • zookeeper_port (Any) (defaults to: 2181)
  • kafka_api_version (Any) (defaults to: '1.0.0')
  • client_id (Any) (defaults to: 'burrow-client')
  • httpserver_port (Any) (defaults to: 8000)
  • lagcheck_intervals (Any) (defaults to: 10)
  • email_template (Any) (defaults to: 'burrow/email.tmpl.erb')
  • consumer_groups_blacklist (Any) (defaults to: '^(console-consumer-|python-kafka-consumer-|test_).*$') 


68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'modules/burrow/manifests/init.pp', line 68

define burrow (
    $zookeeper_hosts,
    $zookeeper_path,
    $kafka_cluster_name,
    $kafka_brokers,
    $alert_whitelist,
    $smtp_server,
    $from_email,
    $to_email,
    $smtp_server_port = 25,
    $kafka_brokers_port = 9092,
    $zookeeper_port = 2181,
    $kafka_api_version='1.0.0',
    $client_id = 'burrow-client',
    $httpserver_port = 8000,
    $lagcheck_intervals = 10,
    $email_template = 'burrow/email.tmpl.erb',
    $consumer_groups_blacklist = '^(console-consumer-|python-kafka-consumer-|test_).*$',
)
{
    ensure_packages('burrow')

    $pidfile_dir = '/var/run/burrow'
    if !defined(File[$pidfile_dir]) {
        file { $pidfile_dir:
            ensure  => 'directory',
            owner   => 'burrow',
            group   => 'burrow',
            mode    => '0755',
            require => Package['burrow'],
        }
    }

    # Burrow 1.0 accepts one parameter named '--config-dir' that
    # expects a directory containing a file named 'burrow.toml'.
    # Since multiple instances of Burrow can run on the same hosts,
    # it is necessary to create the appropriate etc hierarchy.
    $config_dir = "/etc/burrow/${title}"
    file { $config_dir:
        ensure  => 'directory',
    }

    $email_template_path = "${config_dir}/email.tmpl"
    if $to_email {
        file { $email_template_path:
            content => template($email_template),
        }
    }

    file { "${config_dir}/burrow.toml":
        content => template('burrow/burrow.toml.erb'),
    }

    systemd::service { "burrow-${title}":
        ensure    => present,
        content   => systemd_template('burrow'),
        restart   => true,
        subscribe => File["${config_dir}/burrow.toml"],
        require   => [
            Package['burrow'],
            File[$pidfile_dir],
        ],
    }

    if ! defined(Service['burrow']) {
        service { 'burrow':
            ensure  => stopped,
            require => Package['burrow'],
        }
    }
}