You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mark Hamstra <ma...@clearstorydata.com> on 2014/07/09 23:36:43 UTC
ExecutorState.LOADING?
Doesn't look to me like this is used. Does anybody recall what it was
intended for?
Re: ExecutorState.LOADING?
Posted by Aaron Davidson <il...@gmail.com>.
Agreed that the behavior of the Master killing off an Application when
Executors from the same set of nodes repeatedly die is silly. This can also
strike if a single node enters a state where any Executor created on it
quickly dies (e.g., a block device becomes faulty). This prevents the
Application from launching despite only one node being bad.
On Wed, Jul 9, 2014 at 3:08 PM, Mark Hamstra <ma...@clearstorydata.com>
wrote:
> Actually, I'm thinking about re-purposing it. There's a nasty behavior
> that I'll open a JIRA for soon, and that I'm thinking about addressing by
> introducing/using another ExecutorState transition. The basic problem is
> that Master can be overly aggressive in calling removeApplication on
> ExecutorStateChanged. For example, say you have a working, long-running
> Spark stand-alone-mode application and then try to add some more worker
> nodes, but manage to misconfigure the new nodes so that on the new nodes
> Executors never successfully start. In that scenario, you will repeatedly
> end up in the !normalExit branch of Master's receive ExecutorStateChanged,
> quickly exceed ApplicationState.MAX_NUM_RETRY (a non-configurable 10, which
> is another irritation), and end up having your application killed off even
> though it is still running successfully on the old worker nodes.
>
>
>
> On Wed, Jul 9, 2014 at 2:49 PM, Kay Ousterhout <ke...@eecs.berkeley.edu>
> wrote:
>
> > Git history to the rescue! It seems to have been added by Matei way back
> > in July 2012:
> >
> >
> https://github.com/apache/spark/commit/5d1a887bed8423bd6c25660910d18d91880e01fe
> >
> > and then was removed a few months later (replaced by RUNNING) by the same
> > Mr. Zaharia:
> >
> >
> https://github.com/apache/spark/commit/bb1bce79240da22c2677d9f8159683cdf73158c2#diff-776a630ac2b2ec5fe85c07ca20a58fc0
> >
> > So I'd say it's safe to delete it.
> >
> >
> > On Wed, Jul 9, 2014 at 2:36 PM, Mark Hamstra <ma...@clearstorydata.com>
> > wrote:
> >
> > > Doesn't look to me like this is used. Does anybody recall what it was
> > > intended for?
> > >
> >
>
Re: ExecutorState.LOADING?
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Actually, I'm thinking about re-purposing it. There's a nasty behavior
that I'll open a JIRA for soon, and that I'm thinking about addressing by
introducing/using another ExecutorState transition. The basic problem is
that Master can be overly aggressive in calling removeApplication on
ExecutorStateChanged. For example, say you have a working, long-running
Spark stand-alone-mode application and then try to add some more worker
nodes, but manage to misconfigure the new nodes so that on the new nodes
Executors never successfully start. In that scenario, you will repeatedly
end up in the !normalExit branch of Master's receive ExecutorStateChanged,
quickly exceed ApplicationState.MAX_NUM_RETRY (a non-configurable 10, which
is another irritation), and end up having your application killed off even
though it is still running successfully on the old worker nodes.
On Wed, Jul 9, 2014 at 2:49 PM, Kay Ousterhout <ke...@eecs.berkeley.edu>
wrote:
> Git history to the rescue! It seems to have been added by Matei way back
> in July 2012:
>
> https://github.com/apache/spark/commit/5d1a887bed8423bd6c25660910d18d91880e01fe
>
> and then was removed a few months later (replaced by RUNNING) by the same
> Mr. Zaharia:
>
> https://github.com/apache/spark/commit/bb1bce79240da22c2677d9f8159683cdf73158c2#diff-776a630ac2b2ec5fe85c07ca20a58fc0
>
> So I'd say it's safe to delete it.
>
>
> On Wed, Jul 9, 2014 at 2:36 PM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>
> > Doesn't look to me like this is used. Does anybody recall what it was
> > intended for?
> >
>
Re: ExecutorState.LOADING?
Posted by Kay Ousterhout <ke...@eecs.berkeley.edu>.
Git history to the rescue! It seems to have been added by Matei way back
in July 2012:
https://github.com/apache/spark/commit/5d1a887bed8423bd6c25660910d18d91880e01fe
and then was removed a few months later (replaced by RUNNING) by the same
Mr. Zaharia:
https://github.com/apache/spark/commit/bb1bce79240da22c2677d9f8159683cdf73158c2#diff-776a630ac2b2ec5fe85c07ca20a58fc0
So I'd say it's safe to delete it.
On Wed, Jul 9, 2014 at 2:36 PM, Mark Hamstra <ma...@clearstorydata.com>
wrote:
> Doesn't look to me like this is used. Does anybody recall what it was
> intended for?
>