[jira] [Updated] (BIGTOP-1689) puppet: Allow merging arbitrary site configuration

["Rob Leidle (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BIGTOP-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rob Leidle updated BIGTOP-1689:
-------------------------------
    Attachment: BIGTOP-1689.2.patch

Add a project with a merge-config command to support merging in a set of overrides to a Hadoop ecosystem configuration file

This program supports the following types:
* site: Hadoop site file such as core-site.xml, yarn-site.xml, or hbase-site.xml
* env: Hadoop env file such as hadoop-env.sh, yarn-env.sh, or hive-env.sh
* properties: A java properties file such as pig.properties
* sparkconf: The spark conf file format

The program is installed with bigtop-utils RPM and is on the PATH after installation

Puppet manifests have been updated for the hadoop, hive, pig, hbase, and spark modules to make use of merge-config for most of the configuration files in those modules.
Parameters are passed through puppet to merge-config via environment variables to prevent secret configurations such as passwords from being exposed logs.
bigtop_file Puppet modules attempt to minimize the number of merges run as there is a reasonably high cost of executing a JVM.

The pig.properties template is updated to match what is in Pig 0.14 source.

> puppet: Allow merging arbitrary site configuration
> --------------------------------------------------
>
>                 Key: BIGTOP-1689
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1689
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: deployment
>    Affects Versions: 0.8.0
>            Reporter: Peter Slawski
>            Assignee: Peter Slawski
>             Fix For: 1.1.0
>
>         Attachments: BIGTOP-1689.1.patch, BIGTOP-1689.2.patch
>
>
> Puppet should be flexible in allowing arbitrary configuration name value pairs to be merged into a given site.xml file that was generated from a template.
> For example, the following could be included in site.yaml which would add a configuration entry for hadoop.tmp.dir in core-site.xml:
> {code}
> hadoop::common_hdfs::hadoop_core_site_overrides:
>   "hadoop.tmp.dir": "/mnt/var/lib/hadoop/tmp"
> {code}
> This could be implemented as a puppet custom-function taking in the output of template:
> {code}
>     file {
>       "/etc/hadoop/conf/core-site.xml":
>         content => merge_site(template('hadoop/core-site.xml'),  $hadoop_core_site_overrides)
>         require => [Package["hadoop"]],
>     }
> {code}
> Perhaps another approach would be to have site.xml templates be created from a single map of name value pairs. The merge would happen before the file content is generated from the template.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)