You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by colt_colt <we...@hotmail.com> on 2014/03/28 11:25:09 UTC

How does Spark handle executor down? RDD in this executor will be recomputed automatically?

I am curious about Spark fail over scenario, if some executor down,  that
means the JVM crashed. AM will restart the executor, but how about the RDD
data in JVM?  if I didn't persist RDD, does Spark will recompute lost RDD or
just let it lose?  there is some description in Spark site: "Each RDD
remembers the lineage of deterministic operations that were used on a
fault-tolerant input dataset to create it." 

thanks in advance



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-does-Spark-handle-executor-down-RDD-in-this-executor-will-be-recomputed-automatically-tp3422.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: How does Spark handle executor down? RDD in this executor will be recomputed automatically?

Posted by Sonal Goyal <so...@gmail.com>.

Each handle to the RDD holds its lineage information, which means it knows
how it was computed starting from data in a reliable storage or from other
RDDs. RDDs hence can be reconstructed when the node fails.

Best Regards,
Sonal
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>




On Fri, Mar 28, 2014 at 3:55 PM, colt_colt <we...@hotmail.com> wrote:

> I am curious about Spark fail over scenario, if some executor down,  that
> means the JVM crashed. AM will restart the executor, but how about the RDD
> data in JVM?  if I didn't persist RDD, does Spark will recompute lost RDD
> or
> just let it lose?  there is some description in Spark site: "Each RDD
> remembers the lineage of deterministic operations that were used on a
> fault-tolerant input dataset to create it."
>
> thanks in advance
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-does-Spark-handle-executor-down-RDD-in-this-executor-will-be-recomputed-automatically-tp3422.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>