You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Evans Ye (JIRA)" <ji...@apache.org> on 2015/10/21 20:19:27 UTC
[jira] [Comment Edited] (BIGTOP-2074) spark-worker doesn't start
during deploy from master
[ https://issues.apache.org/jira/browse/BIGTOP-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967591#comment-14967591 ]
Evans Ye edited comment on BIGTOP-2074 at 10/21/15 6:18 PM:
------------------------------------------------------------
The bug is fairly complicated. [~Konstantin Boudnik] spots most of them:
* absence of the setting of {{STANDALONE_SPARK_MASTER_HOST}} in the spark-env.sh
* {{export SPARK_MASTER_IP=<%= @master_port %>}}
Another bug I found is that all the code in spark init.pp are lookning for {{$common::master_host}}, but it's {{$common::spark_master_host}} in cluster.yaml instead.
The following is fine since the master_host is set to $fqdn only if {{$spark::common::master_host}} is missing, hence $fqdn should be a good enough default value.
{code}
class common ($master_host = $fqdn, $master_port = "7077", $master_ui_port = "18080")
{code}
I've uploaded a patch to fix the spark worker deployment. It works well on two nodes docker cluster.
was (Author: evans_ye):
The bug is fairly complicated. [~Konstantin Boudnik] spots most of them:
* absence of the setting of {{STANDALONE_SPARK_MASTER_HOST}} in the spark-env.sh
* export SPARK_MASTER_IP=<%= @master_port %>
Another bug I found is that all the code in spark init.pp are lookning for {{$common::master_host}}, but it's {{$common::spark_master_host}} in cluster.yaml instead.
The following is fine since the master_host is set to $fqdn only if {{$spark::common::master_host}} is missing, hence $fqdn should be a good enough default value.
{code}
class common ($master_host = $fqdn, $master_port = "7077", $master_ui_port = "18080")
{code}
I've uploaded a patch to fix the spark worker deployment. It works well on two nodes docker cluster.
> spark-worker doesn't start during deploy from master
> ----------------------------------------------------
>
> Key: BIGTOP-2074
> URL: https://issues.apache.org/jira/browse/BIGTOP-2074
> Project: Bigtop
> Issue Type: Bug
> Components: deployment
> Affects Versions: 1.0.0
> Environment: Official Bigtop ubuntu-14.04 docker image with Hiera 1.3.0
> Reporter: Konstantin Boudnik
> Fix For: 1.1.0
>
> Attachments: BIGTOP-2074.patch
>
>
> spark-worker refuses to start automatically after {{puppet apply}}. The error message is as follows
> {noformat}
> 15/09/26 07:05:54 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@ignite.docker:7078]
> 15/09/26 07:05:54 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkWorker@ignite.docker:7078]
> 15/09/26 07:05:54 INFO util.Utils: Successfully started service 'sparkWorker' on port 7078.
> Exception in thread "main" org.apache.spark.SparkException: Invalid master URL: spark://:7077
> at org.apache.spark.util.Utils$.extractHostPortFromSparkUrl(Utils.scala:1986)
> at org.apache.spark.deploy.master.Master$.toAkkaUrl(Master.scala:879)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)