You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2016/06/02 21:19:59 UTC

[jira] [Created] (SPARK-15736) Gracefully handle loss of cached RDDs' on-disk files

Josh Rosen created SPARK-15736:
----------------------------------

             Summary: Gracefully handle loss of cached RDDs' on-disk files
                 Key: SPARK-15736
                 URL: https://issues.apache.org/jira/browse/SPARK-15736
             Project: Spark
          Issue Type: Bug
          Components: Block Manager
            Reporter: Josh Rosen
            Assignee: Josh Rosen


If an RDD partition is cached on disk and the on-disk file is lost, then reads of that cached partition will fail and the missing partition is supposed to be recomputed by a new task attempt. However, the current behavior is to repeatedly re-attempt the read on the same machine without performing any recomputation, which leads to a complete job failure.

In order to fix this problem, the executor with the missing file needs to properly mark the corresponding block as missing so that it stops advertising itself as a cache location for that block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org