You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bigtop.apache.org by "Amir Sanjar (JIRA)" <ji...@apache.org> on 2017/04/11 18:16:41 UTC
[jira] [Comment Edited] (BIGTOP-2738) spark-worker fails to start
[ https://issues.apache.org/jira/browse/BIGTOP-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15964750#comment-15964750 ]
Amir Sanjar edited comment on BIGTOP-2738 at 4/11/17 6:16 PM:
--------------------------------------------------------------
the issue is caused by change in line 47:
su -s /bin/bash $SVC_USER -c "nohup nice -n 0 \
- ${EXEC_PATH} org.apache.spark.deploy.worker.Worker spark://$SPARK_MASTER_IP:$SPARK_MASTER_PORT $DAEMON_FLAGS \ <<< works
+ ${EXEC_PATH} org.apache.spark.deploy.worker.Worker $SPARK_MASTER_URL $DAEMON_FLAGS \ <<< breaks
> $LOG_FILE 2>&1 & "'echo $!' > "$PIDFILE"
During default installation (i.e apt-get install spark-worker), different spark-env.sh https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/spark/spark-env.sh gets installed.
I recommand to append "export SPARK_MASTER_URL=spark://$SPARK_MASTER_IP:$SPARK_MASTER_PORT" to https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/spark/spark-env.sh, that should fix the problem.
I could submit a patch, but I am at a customer site at the moment.
was (Author: asanjar):
the issue is caused by change in line 47:
su -s /bin/bash $SVC_USER -c "nohup nice -n 0 \
- ${EXEC_PATH} org.apache.spark.deploy.worker.Worker spark://$SPARK_MASTER_IP:$SPARK_MASTER_PORT $DAEMON_FLAGS \ <<< works
+ ${EXEC_PATH} org.apache.spark.deploy.worker.Worker $SPARK_MASTER_URL $DAEMON_FLAGS \ <<< breaks
> $LOG_FILE 2>&1 & "'echo $!' > "$PIDFILE"
During default installation (i.e apt-get install spark-worker), different spark-env.sh https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/spark/spark-env.sh gets installed.
I recommand to append "export SPARK_MASTER_URL=spark://$SPARK_MASTER_IP:$SPARK_MASTER_PORT" to https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/spark/spark-env.sh, that should fix the problem.
> spark-worker fails to start
> ---------------------------
>
> Key: BIGTOP-2738
> URL: https://issues.apache.org/jira/browse/BIGTOP-2738
> Project: Bigtop
> Issue Type: Bug
> Components: spark
> Affects Versions: 1.2.0
> Reporter: Amir Sanjar
> Priority: Blocker
> Fix For: 1.2.0
>
>
> Problem is caused by BIGTOP-2490 that assumes only puppet managed orchestration. Thus, breaks other method of deployment .
> Cause of the problem:
> File bigtop-packages/src/common/spark/spark-worker.svc was changed to look for value $SPARK_MASTER_URL (line 47). This value is defined only in file spark-env.sh (bigtop-deploy/puppet/modules/spark/templates/spark-env.sh) that gets only initialized during puppet deployment.
> https://github.com/apache/bigtop/commit/f627c7868cb1123f6595380a7325f538ebdb8913#diff-90a09a528d21e12a96d209ffc4036f1e
> I propose to remove BIGTOP-2490 and update Bigtop 1.2 repository asap, Hopefully we will have a proper fix soon.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)