Puppet Class: profile::netbox

Defined in:
modules/profile/manifests/netbox.pp

Overview

This profile installs all the Netbox related parts as WMF requires it

Actions:

Deploy Netbox
Install apache, uwsgi
set up Netbox report alerts / automated running.

Examples:

include profile::netbox

Parameters:

  • active_server (Stdlib::Fqdn) (defaults to: lookup('profile::netbox::active_server'))

    the active netbox server

  • service_hostname (Stdlib::Fqdn) (defaults to: lookup('profile::netbox::service_hostname'))

    the fqdn of the service

  • discovery_name (Stdlib::Fqdn) (defaults to: lookup('profile::netbox::discovery_name'))

    The fqdn name used internally

  • additional_sans (Array[Stdlib::Host]) (defaults to: lookup('profile::netbox::additional_sans'))

    A list of fqdn names to be added to the certificate SAN

  • slaves (Array[String]) (defaults to: lookup('profile::netbox::slaves'))

    list of secondary netbox serveres

  • scap_repo (String) (defaults to: lookup('profile::netbox::scap_repo'))

    The repo to use for scap deploys

  • rw_token (String) (defaults to: lookup('profile::netbox::rw_token'))

    api read write token key

  • ro_token (String) (defaults to: lookup('profile::netbox::ro_token'))

    api read only token key

  • dump_interval (String) (defaults to: lookup('profile::netbox::dump_interval'))

    how often to perform dumps

  • db_primary (Stdlib::Fqdn) (defaults to: lookup('profile::netbox::db_primary'))

    primary database name

  • db_password (String) (defaults to: lookup('profile::netbox::db_password'))

    primary database name

  • secret_key (String) (defaults to: lookup('profile::netbox::secret_key'))

    django secret key

  • authentication_provider (Enum['ldap', 'cas']) (defaults to: lookup('profile::netbox::authentication_provider'))

    Either ldap or cas

  • ssl_provider (Profile::Pki::Provider) (defaults to: lookup('profile::netbox::ssl_provider'))

    Either cfssl or acme

  • acme_certificate (Optional[String[1]]) (defaults to: lookup('profile::netbox::acme_cetificate'))

    acme certificate name

  • netbox_api (Stdlib::HTTPSUrl) (defaults to: lookup('profile::netbox::netbox_api'))

    netbox api url

  • ganeti_user (Optional[String]) (defaults to: lookup('profile::netbox::ganeti_user'))

    The ganeti user

  • ganeti_password (Optional[String]) (defaults to: lookup('profile::netbox::ganeti_password'))

    The ganeti password

  • ganeti_sync_interval (Integer) (defaults to: lookup('profile::netbox::ganeti_sync_interval'))

    how frequently to sync with ganeti

  • ganeti_sync_profiles (Array[Profile::Netbox::Ganeti_sync_profile]) (defaults to: lookup('profile::netbox::ganeti_sync_profiles'))

    list of profiles to sync and the config

  • puppetdb_host (Optional[Stdlib::Fqdn]) (defaults to: lookup('profile::netbox::puppetdb_host'))

    the host of the puppetdb server

  • puppetdb_microservice_port (Optional[Stdlib::Port]) (defaults to: lookup('profile::netbox::puppetdb_microservice_port'))

    the port where the puppetd micro service listens

  • report_checks (Array[Profile::Netbox::Report_check]) (defaults to: lookup('profile::netbox::report_checks'))

    a list of report checks

  • librenms_db_user (Optional[String]) (defaults to: lookup('profile::netbox::librenms_db_user'))

    librenms DB user

  • librenms_db_password (Optional[String]) (defaults to: lookup('profile::netbox::librenms_db_password'))

    librenms DB password

  • librenms_db_host (Optional[Stdlib::Fqdn]) (defaults to: lookup('profile::netbox::librenms_db_host'))

    librenms DB host

  • librenms_db_name (Optional[String]) (defaults to: lookup('profile::netbox::librenms_db_name'))

    librenms DB name

  • swift_auth_url (Optional[Stdlib::HTTPUrl]) (defaults to: lookup('profile::netbox::swift_auth_url'))

    Swift auth url

  • swift_user (Optional[String]) (defaults to: lookup('profile::netbox::swift_user'))

    Swift user

  • swift_key (Optional[String]) (defaults to: lookup('profile::netbox::swift_key'))

    Swift key

  • swift_url_key (Optional[String]) (defaults to: lookup('profile::netbox::swift_url_key'))

    Swift url key

  • swift_container (Optional[String]) (defaults to: lookup('profile::netbox::swift_container'))

    Swift container

  • redis_port (Stdlib::Port) (defaults to: lookup('profile::netbox::redis_port'))

    redis port number

  • redis_maxmem (Integer) (defaults to: lookup('profile::netbox::redis_maxmem'))

    redis maximum memory

  • ldap_config (Hash) (defaults to: lookup('ldap', Hash, hash, {}))

    the ldap config for cas

  • do_backups (Boolean) (defaults to: lookup('profile::netbox::do_backup'))

    if we should perform backups

  • http_proxy (Optional[Stdlib::HTTPUrl]) (defaults to: lookup('profile::netbox::http_proxy'))

    proxy server to use for outbound connections

  • changelog_retention (Integer[0]) (defaults to: lookup('profile::netbox::changelog_retention'))

    The number of days to retain logged changes (object creations, updates, and deletions). Set this to 0 to retain changes in the database indefinitely.

  • jobresult_retention (Integer[0]) (defaults to: lookup('profile::netbox::jobresult_retention'))

    The number of days to retain job results (scripts and reports). Set this to 0 to retain job results in the database indefinitely.

  • prefer_ipv4 (Boolean) (defaults to: lookup('profile::netbox::prefer_ipv4'))

    When determining the primary IP address for a device, IPv6 is preferred over IPv4 by default. Set this to True to prefer IPv4 instead.

  • cas_rename_attributes (Hash[String, String]) (defaults to: lookup('profile::netbox::cas_rename_attributes'))

    a mapping of attributes that should be renamed

  • cas_group_attribute_mapping (Hash[String, Array]) (defaults to: lookup('profile::netbox::cas_group_attribute_mapping'))

    a mapping of attributes to netbox groups

  • cas_group_mapping (Hash[String, Array]) (defaults to: lookup('profile::netbox::cas_group_mapping'))

    a mapping of ldap groups to local groups

  • cas_group_required (Array) (defaults to: lookup('profile::netbox::cas_group_required'))

    list of required groups

  • cas_username_attribute (Optional[String]) (defaults to: lookup('profile::netbox::cas_username_attribute'))

    cas attribute to use as a username

  • cas_server_url (Optional[Stdlib::HTTPSUrl]) (defaults to: lookup('profile::netbox::cas_server_url'))

    the location of the cas server



61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
# File 'modules/profile/manifests/netbox.pp', line 61

class profile::netbox (
    Hash                      $ldap_config             = lookup('ldap', Hash, hash, {}),
    Stdlib::Fqdn              $active_server           = lookup('profile::netbox::active_server'),
    Stdlib::Fqdn              $service_hostname        = lookup('profile::netbox::service_hostname'),
    Stdlib::Fqdn              $discovery_name          = lookup('profile::netbox::discovery_name'),
    Array[Stdlib::Host]       $additional_sans         = lookup('profile::netbox::additional_sans'),
    Array[String]             $slaves                  = lookup('profile::netbox::slaves'),
    String                    $scap_repo               = lookup('profile::netbox::scap_repo'),
    String                    $rw_token                = lookup('profile::netbox::rw_token'),
    String                    $ro_token                = lookup('profile::netbox::ro_token'),
    String                    $dump_interval           = lookup('profile::netbox::dump_interval'),
    Stdlib::Fqdn              $db_primary              = lookup('profile::netbox::db_primary'),
    String                    $db_password             = lookup('profile::netbox::db_password'),
    String                    $secret_key              = lookup('profile::netbox::secret_key'),
    Enum['ldap', 'cas']       $authentication_provider = lookup('profile::netbox::authentication_provider'),
    Profile::Pki::Provider    $ssl_provider            = lookup('profile::netbox::ssl_provider'),
    Optional[String[1]]       $acme_certificate        = lookup('profile::netbox::acme_cetificate'),
    Stdlib::HTTPSUrl          $netbox_api              = lookup('profile::netbox::netbox_api'),
    Boolean                   $do_backups              = lookup('profile::netbox::do_backup'),
    Optional[Stdlib::HTTPUrl] $http_proxy              = lookup('profile::netbox::http_proxy'),
    Integer[0]                $changelog_retention     = lookup('profile::netbox::changelog_retention'),
    Integer[0]                $jobresult_retention     = lookup('profile::netbox::jobresult_retention'),
    Boolean                   $prefer_ipv4             = lookup('profile::netbox::prefer_ipv4'),
    Array[Profile::Netbox::Report_check] $report_checks  = lookup('profile::netbox::report_checks'),

    #ganeti config
    Optional[String]           $ganeti_user                 = lookup('profile::netbox::ganeti_user'),
    Optional[String]           $ganeti_password             = lookup('profile::netbox::ganeti_password'),
    Integer                    $ganeti_sync_interval        = lookup('profile::netbox::ganeti_sync_interval'),
    Array[Profile::Netbox::Ganeti_sync_profile] $ganeti_sync_profiles = lookup('profile::netbox::ganeti_sync_profiles'),

    # puppetdb config
    Optional[Stdlib::Fqdn]     $puppetdb_host               = lookup('profile::netbox::puppetdb_host'),
    Optional[Stdlib::Port]     $puppetdb_microservice_port  = lookup('profile::netbox::puppetdb_microservice_port'),


    # Lirenms settings
    Optional[String]           $librenms_db_user            = lookup('profile::netbox::librenms_db_user'),
    Optional[String]           $librenms_db_password        = lookup('profile::netbox::librenms_db_password'),
    Optional[Stdlib::Fqdn]     $librenms_db_host            = lookup('profile::netbox::librenms_db_host'),
    Optional[String]           $librenms_db_name            = lookup('profile::netbox::librenms_db_name'),

    # swift config
    Optional[String]           $swift_user                  = lookup('profile::netbox::swift_user'),
    Optional[String]           $swift_key                   = lookup('profile::netbox::swift_key'),
    Optional[String]           $swift_container             = lookup('profile::netbox::swift_container'),
    Optional[String]           $swift_url_key               = lookup('profile::netbox::swift_url_key'),
    Optional[Stdlib::HTTPUrl]  $swift_auth_url              = lookup('profile::netbox::swift_auth_url'),

    # redis config
    Stdlib::Port               $redis_port                  = lookup('profile::netbox::redis_port'),
    Integer                    $redis_maxmem                = lookup('profile::netbox::redis_maxmem'),


    # CAS Config
    Hash[String, String]       $cas_rename_attributes       = lookup('profile::netbox::cas_rename_attributes'),
    Hash[String, Array]        $cas_group_attribute_mapping = lookup('profile::netbox::cas_group_attribute_mapping'),
    Hash[String, Array]        $cas_group_mapping           = lookup('profile::netbox::cas_group_mapping'),
    Array                      $cas_group_required          = lookup('profile::netbox::cas_group_required'),
    Optional[String]           $cas_username_attribute      = lookup('profile::netbox::cas_username_attribute'),
    Optional[Stdlib::HTTPSUrl] $cas_server_url              = lookup('profile::netbox::cas_server_url'),
) {
    if $ssl_provider == 'acme' and !$acme_certificate {
        fail('must provide \$acme_certificate when using \$ssl_provider acme')
    }
    $ca_certs = '/etc/ssl/certs/ca-certificates.crt'
    # TODO: bring this in from profile::certificates
    $ganeti_ca_cert = '/etc/ssl/certs/Puppet_Internal_CA.pem'
    $puppetdb_api = "https://puppetdb-api.discovery.wmnet:${puppetdb_microservice_port}/"

    $extras_path = '/srv/deployment/netbox-extras/'
    # TODO: Note this prevents overriding other verify options
    # https://github.com/psf/requests/issues/3829
    $systemd_environment = {'REQUESTS_CA_BUNDLE' => '/etc/ssl/certs/ca-certificates.crt'}

    # Used for LDAP auth
    include passwords::ldap::production
    $proxypass = $passwords::ldap::production::proxypass

    # packages required by netbox-extras
    ensure_packages(['python3-git', 'python3-pynetbox', 'python3-requests'])

    # rsyslog forwards json messages sent to localhost along to logstash via kafka
    class { 'profile::rsyslog::udp_json_logback_compat': }
    class { 'netbox':
        service_hostname            => $service_hostname,
        discovery_name              => $discovery_name,
        directory                   => '/srv/deployment/netbox/deploy/src',
        db_host                     => $db_primary,
        db_password                 => $db_password,
        secret_key                  => $secret_key,
        ldap_password               => $proxypass,
        extras_path                 => $extras_path,
        scap_repo                   => $scap_repo,
        swift_auth_url              => $swift_auth_url,
        swift_user                  => $swift_user,
        swift_key                   => $swift_key,
        swift_container             => $swift_container,
        swift_url_key               => $swift_url_key,
        ldap_server                 => $ldap_config['ro-server'],
        authentication_provider     => $authentication_provider,
        local_redis_port            => $redis_port,
        local_redis_maxmem          => $redis_maxmem,
        http_proxy                  => $http_proxy,
        changelog_retention         => $changelog_retention,
        jobresult_retention         => $jobresult_retention,
        prefer_ipv4                 => $prefer_ipv4,
        ca_certs                    => $ca_certs,
        cas_server_url              => $cas_server_url,
        cas_rename_attributes       => $cas_rename_attributes,
        cas_username_attribute      => $cas_username_attribute,
        cas_group_attribute_mapping => $cas_group_attribute_mapping,
        cas_group_mapping           => $cas_group_mapping,
        cas_group_required          => $cas_group_required,
    }
    $ssl_settings = ssl_ciphersuite('apache', 'strong', true)
    class { 'sslcert::dhparam': }
    case $ssl_provider {
        'acme_chief': {
            acme_chief::cert { $acme_certificate:
                puppet_svc => 'apache2',
            }
            # Only use the ec certs this allows us to match cfssl behaviour
            $ssl_paths = {
                'cert'    => "/etc/acmecerts/${acme_certificate}/live/ec-prime256v1.crt",
                'chain'   => "/etc/acmecerts/${acme_certificate}/live/ec-prime256v1.chain.crt",
                'chained' => "/etc/acmecerts/${acme_certificate}/live/ec-prime256v1.chained.crt",
                'key'     => "/etc/acmecerts/${acme_certificate}/live/ec-prime256v1.key",
            }
        }
        'cfssl': {
            $ssl_paths = profile::pki::get_cert('discovery', $service_hostname, {
                'hosts'  => [$facts['networking']['fqdn'], $discovery_name, $additional_sans].flatten.unique,
                'notify' => Service['apache2'],
            })
        }
        default: { fail("unsuupported ssl_provider: ${ssl_provider}") }
    }


    ensure_packages('libapache2-mod-wsgi-py3')
    class { 'httpd':
        modules => ['headers', 'rewrite', 'proxy', 'proxy_http', 'ssl', 'wsgi'],
    }

    ferm::service { 'netbox_https':
        proto => 'tcp',
        port  => '443',
        desc  => 'Public HTTPS for Netbox',
    }

    httpd::site { $service_hostname:
        content => template('profile/netbox/netbox.wikimedia.org.erb'),
    }

    profile::auto_restarts::service { 'apache2': }

    $active_ensure = ($active_server == $facts['networking']['fqdn']).bool2str('present', 'absent')

    # Report Deployment
    #
    # FIXME service::uwsgi seems to create the directory /etc/netbox/, counter intuitively.
    #

    git::clone { 'operations/software/netbox-extras':
        ensure    => 'present',
        directory => $extras_path,
    }

    # Configuration for the Netbox-Ganeti synchronizer
    file { '/etc/netbox/ganeti-sync.cfg':
        owner   => 'netbox',
        group   => 'www-data',
        mode    => '0400',
        content => template('profile/netbox/netbox-ganeti-sync.cfg.erb'),
    }

    # Configuration for Netbox reports in general
    file { '/etc/netbox/reports.cfg':
        owner   => 'netbox',
        group   => 'www-data',
        mode    => '0440',
        content => template('profile/netbox/netbox-reports.cfg.erb'),
    }

    # Configuration for Accounting report which contains secrets
    file { '/etc/netbox/gsheets.cfg':
        owner   => 'netbox',
        group   => 'www-data',
        mode    => '0440',
        content => secret('netbox/gsheets.cfg'),
    }

    # Configurations for the report checker
    file { '/etc/netbox/report_check.yaml':
        owner   => 'root',
        group   => 'nagios',
        mode    => '0440',
        content => to_yaml({
            url   => $netbox_api,
            token => $rw_token,
        }),
    }

    # configurations for other scripts (migrate to this configuration unless
    # there are special needs or permissions are complicated).
    file { '/etc/netbox/scripts.cfg':
        owner   => 'netbox',
        group   => 'www-data',
        mode    => '0440',
        content => template('profile/netbox/netbox-scripts.cfg.erb'),
    }

    # Deploy the report checker
    nrpe::plugin { 'check_netbox_report.py':
        source => 'puppet:///modules/icinga/check_netbox_report.py',
    }

    # Generate report checker icinga checks from Hier data
    $report_checks.each |$report| {
        $repname = $report['name']
        $reportclass = $report['class']

        if $report['alert'] {
            $check_args = ''
        }
        else {
            $check_args = '-w'
        }
        if $report['check_interval'] {
            ::nrpe::monitor_service { "check_netbox_${repname}":
                ensure         => $active_ensure,
                description    => "Netbox report ${repname}",
                nrpe_command   => "/usr/bin/python3 /usr/local/lib/nagios/plugins/check_netbox_report.py ${check_args} ${reportclass}",
                check_interval => $report['check_interval'],
                notes_url      => "https://netbox.wikimedia.org/extras/reports/${reportclass}/",
                contact_group  => 'team-dcops',
            }
        }
        else {
            nrpe::monitor_service { "check_netbox_${repname}":
                ensure    => absent,
                notes_url => 'https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert',
            }
        }
        # This definitely should only be on one of the frontends
        if $report['run_interval'] {
            systemd::timer::job { "netbox_report_${repname}_run":
                ensure          => $active_ensure,
                description     => "Run report ${reportclass} in Netbox",
                environment     => $systemd_environment,
                command         => "/srv/deployment/netbox/venv/bin/python /srv/deployment/netbox/deploy/src/netbox/manage.py runreport ${reportclass}",
                interval        => {
                    'start'    => 'OnCalendar',
                    'interval' => $report['run_interval'],
                },
                logging_enabled => false,
                user            => 'netbox',
            }
        }
        else {
            systemd::timer::job{ "netbox_report_${repname}_run":
                ensure       => absent,
            }
        }
    }

    $ganeti_sync_profiles.each |Integer $prof_index, Hash $profile| {
        systemd::timer::job { "netbox_ganeti_${profile['profile']}_sync":
            ensure                    => $active_ensure,
            description               => "Automatically access Ganeti API at ${profile['profile']} to synchronize to Netbox",
            environment               => $systemd_environment,
            command                   => "/srv/deployment/netbox/venv/bin/python3 /srv/deployment/netbox-extras/tools/ganeti-netbox-sync.py ${profile['profile']}",
            interval                  => {
                'start'    => 'OnCalendar',
                # Splay by 1 minute per profile, offset by 5 minutes from 00 (sync process takes far less than 1 minute)
                'interval' => '*-*-* *:%02d/%02d:00'.sprintf($prof_index + 5, $ganeti_sync_interval),
            },
            logging_enabled           => false,
            monitoring_enabled        => true,
            monitoring_contact_groups => 'admins',
            user                      => 'netbox',
        }
    }

    # Support directory for dumping tables
    file { '/srv/netbox-dumps/':
        ensure => 'directory',
        owner  => 'netbox',
        group  => 'netbox',
        mode   => '0770',
    }
    # Timer for dumping tables
    systemd::timer::job { 'netbox_dump_run':
        ensure                    => $active_ensure,
        description               => 'Dump CSVs from Netbox.',
        environment               => $systemd_environment,
        command                   => '/srv/deployment/netbox-extras/tools/rotatedump',
        interval                  => {
            'start'    => 'OnCalendar',
            'interval' => $dump_interval,
        },
        logging_enabled           => false,
        monitoring_enabled        => false,
        monitoring_contact_groups => 'admins',
        user                      => 'netbox',
    }

    if $do_backups {
      include profile::backup::host
      backup::set { 'netbox': }
    }
}