You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by preeze <et...@gmail.com> on 2014/10/17 12:51:55 UTC
Designed behavior when master is unreachable.
Hi all,
I am running a standalone spark cluster with a single master. No HA or
failover is configured explicitly (no ZooKeeper etc).
What is the default designed behavior for submission of new jobs when a
single master went down or became unreachable?
I couldn't find it documented anywhere.
Thanks.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Designed-behavior-when-master-is-unreachable-tp16677.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Designed behavior when master is unreachable.
Posted by preeze <et...@gmail.com>.
Hi Andrew,
The behavior that I see now is that under the hood it tries to reconnect
endlessly. While this lasts, the thread that tries to fire a new task is
blocked at JobWaiter.awaitResult() and never gets released.
The full stacktrace for spark-1.0.2 is:
"jmsContainer-7" prio=10 tid=0x00007f18fc05b800 nid=0x6250 in Object.wait()
[0x00007f18bdb4b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.$$YJP$$wait(Native Method)
at java.lang.Object.wait(Object.java)
at java.lang.Object.wait(Object.java:503)
at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73) -
locked <0x00000007e735d3b0> (a org.apache.spark.scheduler.JobWaiter)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:452)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1051)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1069)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1083)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1097)
at org.apache.spark.rdd.RDD.foreachPartition(RDD.scala:724)
...
I observed the same behavior on both spark-1.0.2 and spark-1.1.0.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Designed-behavior-when-master-is-unreachable-tp16677p16819.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Designed behavior when master is unreachable.
Posted by Andrew Ash <an...@andrewash.com>.
I'm not sure what the design is, but I think the current behavior if the
driver can't reach the master is to attempt to connect once and fail if
that attempt fails. Is that what you're observing? (What version of Spark
also?)
On Fri, Oct 17, 2014 at 3:51 AM, preeze <et...@gmail.com> wrote:
> Hi all,
>
> I am running a standalone spark cluster with a single master. No HA or
> failover is configured explicitly (no ZooKeeper etc).
>
> What is the default designed behavior for submission of new jobs when a
> single master went down or became unreachable?
>
> I couldn't find it documented anywhere.
> Thanks.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Designed-behavior-when-master-is-unreachable-tp16677.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>