[jira] [Comment Edited] (BIGTOP-1746) Introduce the concept of roles in bigtop cluster deployment

["Richard Pelavin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355716#comment-14355716 ] 

Richard Pelavin edited comment on BIGTOP-1746 at 3/10/15 9:10 PM:
------------------------------------------------------------------

Vishnu,

Good comments. I think useful to divide into two issues:

1) having a simple topology representation that is more intuitive and neutral; one that resonates with those who dont know hiera, vagrant, etc

2) having such a representation that has the flexibility to capture a wide variety of topologies

Think if 1 is implemented we could write simple code that "compiled" the topology into hiera/vagrant, etc and then power by existing bigtop instrumentation/mechanism.
 
I think your comments are centered on '2'. The way I would handle this is as follows, which is predicated on whether there is a desire in bigtop to move in the direction of providing more sophisticated deployment and orchestration capabilities:

An approach is to flesh out a rich topology that we would eventually want to instrument, but we could view this as an internal spec and then just expose/support a subset of the topologies that are instrumented currently by bigtop, such as a limited set of topologies that have a single master and set of slaves. So in a way the rich topology serves as a roadmap to future extensions and can help capture in advance how potential extensions to treat a richer set of topologies can be modularly folded in.


Rich



was (Author: rnp):
Vishnu,

Good comments. I think useful to divide into two issues:

1) having a simple topology representation that is more intuitive and neutrall one that resonates with those who dont know hiera, vagrant, etc

2) having such a representation that has the flexibility to capture a wide variety of topologies

Think if 1 is implemented we could write simple code that "compiled" the topology into hiera/vagrant, etc and then power by existing bigtop instrumentation/mechanism.
 
I think your comments are centered on '2'. The way I would handle this is as follows, which is predicated on whether there is a desire in bigtop to move in the direction of providing more sophisticated deployment and orchestration capabilities:

An approach is to flesh out a rich topology that we would eventually want to instrument, but we could view this as an internal spec and then just expose/support a subset of the topologies that are instrumented currently by bigtop, such as a limited set of topologies that have a single master and set of slaves. So in a way the rich topology serves as a roadmap to future extensions and can help capture in advance how potential extensions to treat a richer set of topologies can be modularly folded in.


- Rich


> Introduce the concept of roles in bigtop cluster deployment
> -----------------------------------------------------------
>
>                 Key: BIGTOP-1746
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1746
>             Project: Bigtop
>          Issue Type: New Feature
>          Components: deployment
>            Reporter: vishnu gajendran
>              Labels: features
>             Fix For: 0.9.0
>
>
> Currently, during cluster deployment, puppet categorizes nodes as head_node, worker_nodes, gateway_nodes, standy_node based on user specified info. This functionality gives user control over picking up a particular node as head_node, standy_node, gateway_node and rest others as worker_nodes. But, I woulld like to have more fine-grained control on which deamons should run on which node. For example, I do not want to run namenode, datanode on the same node. This functionality can be introduced with the concept of roles. Each node can be assigned a set of role. For example, Node A can be assigned ["namenode", "resourcemanager"] roles. Node B can be assigned ["datanode", "nodemanager"] and Node C can be assigned ["nodemanager", "hadoop-client"]. Now, each node will only run the specified daemons. Prerequisite for this kind of deployment is that each node should be given the necessary configurations that it needs to know. For example, each datanode should know which is the namenode etc... This functionality will allow users to customize the cluster deployment according to their needs. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)