Puppet Class: conda_analytics

Defined in:
modules/conda_analytics/manifests/init.pp

Overview

Class conda_analytics

Installs the conda-analytics.deb package. This package contains a conda environment with python and spark and other packages installed that are useful for working in the WMF Analytics data lake.

Parameters

[ensure] This parameter determines whether or not to install the conda-analytics-package.

[ensure_next] This parameter determines whether or not to install the conda-analytics-next package, which allows for the concurrent installation of two environments. The intention is that the -next package should be available to help with the migration of workload from one spark version to another.

[remove_conda_env_pkgs_dir] If true, a conda-analytics/remove-pkgs debconf setting will be set to true. This setting is used by the conda-analytics .deb postinst step to determine if the conda pkgs/ dir should be removed after installing the packages. Conda pkgs/ dir is only needed if you ever plan to 'clone' the conda env, which is only done on client nodes by real users. Worker nodes can have pkgs/ removed to save space. Default: true

Parameters:

  • ensure (Wmflib::Ensure) (defaults to: 'present')
  • ensure_next (Wmflib::Ensure) (defaults to: 'absent')
  • remove_conda_env_pkgs_dir (Boolean) (defaults to: true) 


28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
 
# File 'modules/conda_analytics/manifests/init.pp', line 28

class conda_analytics(
    Wmflib::Ensure $ensure = 'present',
    Wmflib::Ensure $ensure_next = 'absent',
    Boolean $remove_conda_env_pkgs_dir = true,
) {
    package { 'conda-analytics':
        ensure => stdlib::ensure($ensure, 'package'),
    }

    package { 'conda-analytics-next':
        ensure => stdlib::ensure($ensure_next, 'package'),
    }

    # This is where the conda-analytics .deb package will install the conda-analytics conda environment.
    # Set this variable here for users of this class to have a reference to this.
    $prefix = '/opt/conda-analytics'

    # This is where the conda-analytics-next .deb package will install the conda-analytics-next conda environment.
    # Set this variable here for users of this class to have a reference to this.
    $prefix_next = '/opt/conda-analytics-next'

    # TODO: Remove this conditional and use $remove_conda_env_pkgs_dir as the value of the setting.
    # For this to work, the postinst script has to be smarter and actually check or this value.
    # See: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/blob/main/.gitlab-ci.yml
    if $remove_conda_env_pkgs_dir {
        $conda_packages = $ensure_next ? {
            true    => ['conda-analytics', 'conda-analytics-next'],
            default => ['conda-analytics'],
        }

        $conda_packages.each |String $pkg| {
            debconf::set { "${pkg}/remove-pkgs":
                owner  => $pkg,
                type   => 'boolean',
                value  => true,
                before => Package[$pkg],
            }
        }
    }
}