[jira] [Comment Edited] (BIGTOP-1746) Introduce the concept of roles in bigtop cluster deployment

["Evans Ye (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-1746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699935#comment-14699935 ] 

Evans Ye edited comment on BIGTOP-1746 at 8/17/15 5:56 PM:
-----------------------------------------------------------

Hey, [~vishnu]. First of all, thanks for staying with me on this. If you find something need to discuss, just post your thought here. We'd love to discuss. There's no need to provide patch in hurry. :)
My two cents, might not be the best: 
* Instead of using the configuration {{hadoop-hive::client::use_tez: true}}, which is to change the hive execution engine value to tez, how about we explicitly specify it as {{hadoop-hive::client::exec_engine: tez}}? Just like what you did for {{mapreduce_framework_name}}.
* It looks like the value of {{hadoop::common::use_tez}} is used to decide whether or not to add tez libs into HADOOP_CLASSPATH, which is fine. But this brings up another issue I can recall that we've discussed in [mailing list|https://mail-archives.apache.org/mod_mbox/bigtop-dev/201505.mbox/%3C20150525104146.GN8680@tpx%3E]
We haven't reach a conclusion yet, but the truth is, we're using different practice for different packages. Take iginte as an example, it directly smylink iginte libs into hadoop/libs so that adding classpath is no longer needed:
{code}
[root@bigtop1 lib]# ll /usr/lib/hadoop/lib |grep ignite
lrwxrwxrwx 1 root root      43 Aug 17 16:15 ignite-core.jar -> /usr/lib/ignite-hadoop/libs/ignite-core.jar
lrwxrwxrwx 1 root root      59 Aug 17 16:15 ignite-hadoop.jar -> /usr/lib/ignite-hadoop/libs/ignite-hadoop/ignite-hadoop.jar
{code}

At this moment I think we should decide which practice we're going for. Does anyone have a quick answer how the other deployment system do? 
Anyone also has opinions to the patch? Probably [~michaelweiser], [~rnp], [~rleidle] can add some color.


was (Author: evans_ye):
Hey, [~vishnu gajendran]. First of all, thanks for staying with me on this. If you find something need to discuss, just post your thought here. We'd love to discuss. There's no need to provide patch in hurry. :)
My two cents, might not be the best: 
* Instead of using the configuration {{hadoop-hive::client::use_tez: true}}, which is to change the hive execution engine value to tez, how about we explicitly specify it as {{hadoop-hive::client::exec_engine: tez}}? Just like what you did for {{mapreduce_framework_name}}.
* It looks like the value of {{hadoop::common::use_tez}} is used to decide whether or not to add tez libs into HADOOP_CLASSPATH, which is fine. But this brings up another issue I can recall that we've discussed in [mailing list|https://mail-archives.apache.org/mod_mbox/bigtop-dev/201505.mbox/%3C20150525104146.GN8680@tpx%3E]
We haven't reach a conclusion yet, but the truth is, we're using different practice for different packages. Take iginte as an example, it directly smylink iginte libs into hadoop/libs so that adding classpath is no longer needed:
{code}
[root@bigtop1 lib]# ll /usr/lib/hadoop/lib |grep ignite
lrwxrwxrwx 1 root root      43 Aug 17 16:15 ignite-core.jar -> /usr/lib/ignite-hadoop/libs/ignite-core.jar
lrwxrwxrwx 1 root root      59 Aug 17 16:15 ignite-hadoop.jar -> /usr/lib/ignite-hadoop/libs/ignite-hadoop/ignite-hadoop.jar
{code}

At this moment I think we should decide which practice we're going for. Does anyone have a quick answer how the other deployment system do? 
Anyone also has opinions to the patch? Probably [~michaelweiser], [~rnp], [~Rob Leidle] can add some color.

> Introduce the concept of roles in bigtop cluster deployment
> -----------------------------------------------------------
>
>                 Key: BIGTOP-1746
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-1746
>             Project: Bigtop
>          Issue Type: New Feature
>          Components: deployment
>    Affects Versions: 0.8.0
>            Reporter: vishnu gajendran
>            Assignee: vishnu gajendran
>              Labels: features
>             Fix For: 1.1.0
>
>         Attachments: BIGTOP-1746.patch, BIGTOP-1746.patch, BIGTOP-1746.patch, BIGTOP-1746.patch
>
>
> Currently, during cluster deployment, puppet categorizes nodes as head_node, worker_nodes, gateway_nodes, standy_node based on user specified info. This functionality gives user control over picking up a particular node as head_node, standy_node, gateway_node and rest others as worker_nodes. But, I woulld like to have more fine-grained control on which deamons should run on which node. For example, I do not want to run namenode, datanode on the same node. This functionality can be introduced with the concept of roles. Each node can be assigned a set of role. For example, Node A can be assigned ["namenode", "resourcemanager"] roles. Node B can be assigned ["datanode", "nodemanager"] and Node C can be assigned ["nodemanager", "hadoop-client"]. Now, each node will only run the specified daemons. Prerequisite for this kind of deployment is that each node should be given the necessary configurations that it needs to know. For example, each datanode should know which is the namenode etc... This functionality will allow users to customize the cluster deployment according to their needs. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)