[jira] [Commented] (BIGTOP-2764) deployment failure when roles include spark::common and spark::yarn*

["Kevin W Monroe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BIGTOP-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16004857#comment-16004857 ] 

Kevin W Monroe commented on BIGTOP-2764:
----------------------------------------

This is caused by the unfortunate requirement to include datanucleus for all spark deployments (see BIGTOP-2268).

My proposal is to one of the following:

# spend some time on BIGTOP-2268 so hive/datanucleus is properly untangled from spark, and remove [the hack|https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/spark/manifests/init.pp#L154].
# temporarily remove the datanucleus inclusion from [yarn|https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/spark/manifests/init.pp#L117] and [yarn_slave|https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/spark/manifests/init.pp#L122] since we know datanucleus will already be provided by the {{common}} class. 

For reference, here's my {{site.yaml}} that triggers this failure:

{code}
$ cat ./bigtop.release/bigtop-1.2.0/bigtop-deploy/puppet/hieradata/site.yaml
# Juju manages this file. Modifications may be overwritten!
bigtop::bigtop_repo_uri: http://bigtop-repos.s3.amazonaws.com/releases/1.2.0/ubuntu/16.04/x86_64
bigtop::hadoop_head_node: machine-1.qnvvwf5bfm5uzgvdpofqo5megg.dx.internal.cloudapp.net
bigtop::jdk_preinstalled: true
bigtop::roles:
- hadoop-client
- spark-client
- spark-history-server
- spark-master
- spark-on-yarn
- spark-worker
- spark-yarn-slave
bigtop::roles_enabled: true
hadoop::common_hdfs::hadoop_namenode_host: machine-1.qnvvwf5bfm5uzgvdpofqo5megg.dx.internal.cloudapp.net
hadoop::common_hdfs::namenode_datanode_registration_ip_hostname_check: false
hadoop::common_mapred_app::jobtracker_host: machine-1.qnvvwf5bfm5uzgvdpofqo5megg.dx.internal.cloudapp.net
hadoop::common_mapred_app::mapreduce_jobhistory_host: machine-1.qnvvwf5bfm5uzgvdpofqo5megg.dx.internal.cloudapp.net
hadoop::common_yarn::hadoop_ps_host: machine-1.qnvvwf5bfm5uzgvdpofqo5megg.dx.internal.cloudapp.net
hadoop::common_yarn::hadoop_rm_host: machine-1.qnvvwf5bfm5uzgvdpofqo5megg.dx.internal.cloudapp.net
hadoop::common_yarn::yarn_nodemanager_vmem_check_enabled: false
hadoop::hadoop_storage_dirs:
- /data/1
- /data/2
spark::common::event_log_dir: hdfs:///var/log/spark/apps
spark::common::history_log_dir: hdfs:///var/log/spark/apps
spark::common::master_host: 192.168.0.4
spark::common::master_url: yarn-client
spark::common::zookeeper_connection_string: null
{code}

> deployment failure when roles include spark::common and spark::yarn*
> --------------------------------------------------------------------
>
>                 Key: BIGTOP-2764
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-2764
>             Project: Bigtop
>          Issue Type: Bug
>          Components: deployment
>    Affects Versions: 1.2.0
>            Reporter: Kevin W Monroe
>
> I use bigtop roles to control what gets deployed.  When I include `spark-client` and `spark-on-yarn`, I get the following deployment failure:
> {code}
> unit-spark-0: 15:15:24 INFO unit.spark/0.config-changed Error: Evaluation Error: Error while evaluating a Resource Statement, Duplicate declaration: Package[spark-datanucleus] is already declared in file /home/ubuntu/bigtop.release/bigtop-1.2.0/bigtop-deploy/puppet/modules/spark/manifests/init.pp:158; cannot redeclare at /home/ubuntu/bigtop.release/bigtop-1.2.0/bigtop-deploy/puppet/modules/spark/manifests/init.pp:132 at /home/ubuntu/bigtop.release/bigtop-1.2.0/bigtop-deploy/puppet/modules/spark/manifests/init.pp:132:5 on node machine-0.qnvvwf5bfm5uzgvdpofqo5megg.dx.internal.cloudapp.net
> {code}
> This is because the {{spark-datanucleus}} package is defined for both the common class:
> https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/spark/manifests/init.pp#L158
> and the yarn classes:
> https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/modules/spark/manifests/init.pp#L117



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)