You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2014/11/06 20:53:34 UTC
[jira] [Commented] (SPARK-4280) In dynamic allocation, add option
to never kill executors with cached blocks
[ https://issues.apache.org/jira/browse/SPARK-4280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200785#comment-14200785 ]
Andrew Or commented on SPARK-4280:
----------------------------------
Hey [~sandyr] does this require the application to explicitly uncache the RDDs? My concern is that we cache some blocks behind the application's back (e.g. broadcast, streaming blocks) and we don't currently display these on the UI, in which case the application will never remove executors. It seems that some mechanism that would unconditionally blow away all the blocks on an executor will be handy.
> In dynamic allocation, add option to never kill executors with cached blocks
> ----------------------------------------------------------------------------
>
> Key: SPARK-4280
> URL: https://issues.apache.org/jira/browse/SPARK-4280
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Sandy Ryza
>
> Even with the external shuffle service, this is useful in situations like Hive on Spark where a query might require caching some data. We want to be able to give back executors after the job ends, but not during the job if it would delete intermediate results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org