How Spark handles dead machines during a job.

[Sung Hwan Chung <co...@gmail.com>]

Hello,

Say, that I'm doing a simple rdd.map followed by collect. Say, also, that
one of the executors finish all of its tasks, but there are still other
executors running.

If the machine that hosted the finished executor gets terminated, does the
master still have the results from the finished tasks (and thus doesn't
restart those finished tasks)?

Or does the master require that all the executors be alive during the
entire map-collect cycle?

Thanks!

Re: How Spark handles dead machines during a job.

[Reynold Xin <rx...@databricks.com>]

The driver has the data and wouldn't need to rerun.

On Friday, April 8, 2016, Sung Hwan Chung <co...@gmail.com> wrote:

> Hello,
>
> Say, that I'm doing a simple rdd.map followed by collect. Say, also, that
> one of the executors finish all of its tasks, but there are still other
> executors running.
>
> If the machine that hosted the finished executor gets terminated, does the
> master still have the results from the finished tasks (and thus doesn't
> restart those finished tasks)?
>
> Or does the master require that all the executors be alive during the
> entire map-collect cycle?
>
> Thanks!
>

Re: How Spark handles dead machines during a job.

[Reynold Xin <rx...@databricks.com>]

The driver has the data and wouldn't need to rerun.

On Friday, April 8, 2016, Sung Hwan Chung <co...@gmail.com> wrote:

> Hello,
>
> Say, that I'm doing a simple rdd.map followed by collect. Say, also, that
> one of the executors finish all of its tasks, but there are still other
> executors running.
>
> If the machine that hosted the finished executor gets terminated, does the
> master still have the results from the finished tasks (and thus doesn't
> restart those finished tasks)?
>
> Or does the master require that all the executors be alive during the
> entire map-collect cycle?
>
> Thanks!
>