works disconnected with master but still keep alive

[Cheney Sun <su...@gmail.com>]

Hi experts,

I set up an Spark cluster in the standalone mode with 10 workers and the
version is 0.9.1. I chose the version with the assumption that the latest
version is always the most stable one. However, when I unintentionally run
an problematic job (such as config the SPARK_HOME with a wrong path), it
will make the workers disconnected with master, and master reject the
re-register of workers. In fact, the worker process didn't die and the
worker port is still normally open, which make the situation worse for an
online system because the monitor system (nagios) can't report the issue.

Before I turned to the mail list, I did some quick searches and found
someone had already reported the same issue (follow the links below). Can
anyone confirm if it's a known bug? If so, is there any fix for it? Or can
I work around it through some way?

http://stackoverflow.com/questions/22031006/spark-0-9-0-worker-keeps-dying-in-standalone-mode-when-job-fails
http://apache-spark-user-list.1001560.n3.nabble.com/master-attempted-to-re-register-the-worker-and-then-took-all-workers-as-unregistered-td553.html
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Worker-crashing-and-Master-not-seeing-recovered-worker-td2312.html

Thanks a lot,
Cheney

Re: works disconnected with master but still keep alive

[Cheney Sun <su...@gmail.com>]

No reply, maybe I didn't make it clear. I try to add more information.

When the worker node attempts to launch a problematic executor, not only the
executor fails to launch but also the worker is removed by master. The
worker will try to re-register with master but rejected. In the master log,
the following WARN message continually pops up.

14/05/05 09:55:00 WARN Master: Got heartbeat from unregistered worker
worker-20140504213545-spark-host007-7078

Can anyone shed some lights on the issue? 



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/works-disconnected-with-master-but-still-keep-alive-tp5315p5338.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.