[jira] [Comment Edited] (BIGTOP-1047) Support Puppet 3.x

["Richard Pelavin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14056728#comment-14056728 ] 

Richard Pelavin edited comment on BIGTOP-1047 at 7/9/14 8:43 PM:
-----------------------------------------------------------------

As a first step in the task of refactoring the puppet modules to handle puppet >=3.x, wanted to start with a design approach for handling attribute settings. Key goal is using defaults, but to make it easy to override any attribute that can appear in any hadoop config file. The current puppet implementation had a nice way to deal with this using the Puppet extlookup function.  Changes are needed to handle Puppet >=3.x which no longer supports dynamic scoping and to take advantage of Puppet functionality, such as Hiera that provides more convenient capabilities. Will shortly propose an approach that takes the Hiera design pattern and will support Hiera, but also will support other representations that encode the same information.

The approach I am proposing to embark on is to separate this into a number of sub-task


-    Propose a global namespace for describing hadoop config attributes that will be applicable to a wide range of hadoop topologies with various mixtures of services; I will look through all the related work in Bigtop and leverage anything done here already.

-    Put in a simple mechanism that better integrates 
     -- the way Linux packages include reference config files, and 
     -- templates (e.g Erubis, Jinja2) used by config management systems 
that tend to override and ignore what is packaged in the Linux package. This seems like a unique opportunity for a project that handles both packaging and configuration to better align the two. I will elaborate shortly, but this would entail putting inside the packages themselves either templates or a simple meta description of what parameters can be possibly set and their defaults. An important consideration is making this so it can be incrementally and gracefully folded in; so a key design goal would be to handle for example just certain config fragments that are typically parametrized and falling back on existing approach of having a sample config files with defaults that you copy over.  As a reference point will look at Augeas, which has both some successes and some challenges

-    Implement using this approach first HDFS and refine approach before moving on to the other services. Yarn we be treated second.


was (Author: rnp):
As a first step in the task of refactoring the puppet modules to handle puppet >=3.x, wanted to start with a design approach for handling attribute settings. Key goal is using defaults, but to make it easy to override any attribute that can appear in any hadoop config file. The current puppet implementation had a nice way to deal with this using the Puppet extlookup function.  Changes are needed to handle Puppet >=3.x which no longer supports dynamic scoping and to take advantage of Puppet functionality, such as Hiera that provides more convenient capabilities. Will shortly propose an approach that takes the Hiera design pattern and will support Hiera, but also will support other representations that encode the same information.

The approach I am proposing to embark on is to separate this into a number of sub-task


-    Propose a global namespace for describing hadoop config attributes that will be applicable to a wide range of hadoop topologies with various mixtures of services; I will look through all the related work in Bigtop and leverage anything done here already.

-    Put in a simple mechanism that better integrates 
     - the way Linux packages include reference config files, and 
     - templates (e.g Erubis, Jinja2) used by config management systems 
that tend to override and ignore what is packaged in the Linux package. This seems like a unique opportunity for a project that handles both packaging and configuration to better align the two. I will elaborate shortly, but this would entail putting inside the packages themselves either templates or a simple meta description of what parameters can be possibly set and their defaults. An important consideration is making this so it can be incrementally and gracefully folded in; so a key design goal would be to handle for example just certain config fragments that are typically parametrized and falling back on existing approach of having a sample config files with defaults that you copy over.  As a reference point will look at Augeas, which has both some successes and some challenges

-    Implement using this approach first HDFS and refine approach before moving on to the other services. Yarn we be treated second.

> Support Puppet 3.x
> ------------------
>
>                 Key: BIGTOP-1047
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1047
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: Deployment
>    Affects Versions: 0.6.0
>            Reporter: Andrey Klochkov
>         Attachments: BIGTOP-1047--n2.patch, BIGTOP-1047.patch
>
>
> Currently bigtop-deploy/puppet module supports Puppet 2.x only. In particular, this is because of using dynamic scoping which was deprecated since 2.5 and not supported in 3.x. This task is to get rid of dynamic scoping to make bigtop-deploy/puppet working with Puppet 3.x.
> More on dynamic scoping in Puppet: http://docs.puppetlabs.com/guides/scope_and_puppet.html



--
This message was sent by Atlassian JIRA
(v6.2#6252)