[jira] [Created] (SPARK-9197) Cached RDD partitions are lost when executors are dynamically deallocated

["Ryan Williams (JIRA)" <ji...@apache.org>]

Ryan Williams created SPARK-9197:
------------------------------------

             Summary: Cached RDD partitions are lost when executors are dynamically deallocated
                 Key: SPARK-9197
                 URL: https://issues.apache.org/jira/browse/SPARK-9197
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 1.4.1
            Reporter: Ryan Williams


Currently, dynamic allocation cleans up executors that have not run any tasks for a certain amount of time.

However, this often leads to cached RDD partitions being lost.

Should dynamic allocation leave executors alone that have cached partitions? Should this be configurable?

Is there any interest in code that would shuffle cached partitions around in preparation for executor-deallocation, to avoid this? Such logic could be useful in general for maintaining persisted RDDs across executor churn.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org