Defined Type: cdh::hadoop::directory

Defined in:
modules/cdh/manifests/hadoop/directory.pp

Overview

Define cdh::hadoop::directory

Creates or removes a directory in HDFS.

Notes:

This will not check ownership and permissions of a directory. It will only check for the directory's existence. If it does not exist, the directory will be created and given specified ownership and permissions. This will not attempt to set ownership and permissions if the directory already exists.

This define does not support managing files in HDFS, only directories.

Ideally this define would be ported into a Puppet File Provider. I once spent some time trying to make that work, but it was more difficult than it sounds. For example, you'd need to handle conversion between symbolic mode to numeric mode, as I could not find a way to get hdfs dfs to list numeric modes for comparison. Perhaps there's a way to use HttpFS to do this instead?

Parameters:

$path - HDFS directory path. Default: $title $ensure - present|absent. Default: present $owner - HDFS directory owner. Default: hdfs $group - HDFS directory group owner. Default: hdfs $mode - HDFS directory mode. Default 0755 $use_kerberos - Configure whether Secure HDFS is used or not. Default: false

If $use_kerberos is enabled, a wrapper is launched which
gets a ticket for the service principal via a keytab. This
wrapper is not part of the cdh module, but can be found at
https://github.com/wikimedia/puppet/modules/kerberos.

Parameters:

  • path (Any) (defaults to: $title)
  • ensure (Any) (defaults to: 'present')
  • owner (Any) (defaults to: 'hdfs')
  • group (Any) (defaults to: 'hdfs')
  • mode (Any) (defaults to: '0755')
  • use_kerberos (Any) (defaults to: false) 


35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
 
# File 'modules/cdh/manifests/hadoop/directory.pp', line 35

define cdh::hadoop::directory (
    $path         = $title,
    $ensure       = 'present',
    $owner        = 'hdfs',
    $group        = 'hdfs',
    $mode         = '0755',
    $use_kerberos = false,
)
{
    Class['cdh::hadoop'] -> Cdh::Hadoop::Directory[$title]

    if $ensure == 'present' {
        kerberos::exec { "cdh::hadoop::directory ${title}":
            command      => "/usr/bin/hdfs dfs -mkdir -p ${path} && /usr/bin/hdfs dfs -chmod ${mode} ${path} && /usr/bin/hdfs dfs -chown ${owner}:${group} ${path}",
            unless       => "/usr/bin/hdfs dfs -test -e ${path}",
            user         => 'hdfs',
            timeout      => 30,
            use_kerberos => $use_kerberos,
        }
    }
    else {
        kerberos::exec { "cdh::hadoop::directory ${title}":
            command      => "/usr/bin/hdfs dfs -rm -R -skipTrash ${path}",
            onlyif       => "/usr/bin/hdfs dfs -test -e ${path}",
            user         => 'hdfs',
            require      => Service['hadoop-hdfs-namenode'],
            timeout      => 30,
            use_kerberos => $use_kerberos,
        }
    }
}