You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Roberto Vaquerizo Rodriguez (JIRA)" <ji...@apache.org> on 2015/01/09 09:41:34 UTC
[jira] [Created] (SPARK-5171) Standalone cluster: masters
scheduling independently
Roberto Vaquerizo Rodriguez created SPARK-5171:
--------------------------------------------------
Summary: Standalone cluster: masters scheduling independently
Key: SPARK-5171
URL: https://issues.apache.org/jira/browse/SPARK-5171
Project: Spark
Issue Type: Bug
Components: Deploy
Affects Versions: 1.2.0
Reporter: Roberto Vaquerizo Rodriguez
Hi.
I'm trying to configure a spark standalone cluster (release = spark 1.2), with three master nodes (bigdata1, bigdata2 and bigdata3) managed by Zookeeper.
It seems there's a configuration problem, since every master node is saying it is the cluster leader:
.........
14/12/30 13:54:59 INFO Master: I have been elected leader! New state: ALIVE
The message above is dumped by every master I start up.
Zookeeper is configured identically in all of them, as follows:
........
dataDir=/spark
........
The only difference is the "myid" file in the /spark directory, of course.
The masters are started using the following configuration:
.........
export SPARK_DAEMON_JAVA_OPTS=" \
-Dspark.deploy.recoverymode=ZOOKEEPER \
-Dspark.deploy.zookeeper.url=bigdata1:2181,bigdata2:2181,bigdata3:2181"
I have not set the spark.deploy.zookeeper.dir variable, since I'm using the default value, /spark, configured in zookeeper, as I mentioned before.
I would like to know if there is any other thing I have to configure, in order to make the masters behave correctly (only one master node active at a time, while the others are in stand-by mode).
With the current situation, I can connect workers and applications to the whole cluster. For instance, I can connect a worker to the cluster using:
spark-class org.apache.spark.deploy.worker.Worker spark://bigdata1:2181,bigdata2:2181,bigdata3:2181
But the worker gets registered to each of the masters independently.
If I stop one of the masters, it tries to re-register to it.
The notion of only one active master is completely lost. All masters are scheduling independently.
If I run "zkServer.sh status" in the cluster, only one of the master nodes says "leader", and the other ones say "follower". So, from a Zookeeper point of view, it seems that the configuration is correct.
But the standalone cluster is not interacting with zookeeper properly.
Do you have any idea?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org