You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Guoqiang Li (JIRA)" <ji...@apache.org> on 2014/07/19 21:09:38 UTC
[jira] [Updated] (SPARK-2595) The driver run garbage collection,
when the executor throws OutOfMemoryError exception
[ https://issues.apache.org/jira/browse/SPARK-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Guoqiang Li updated SPARK-2595:
-------------------------------
Description:
[SPARK-1103|https://issues.apache.org/jira/browse/SPARK-1103] implementation GC-based cleaning only consider the memory usage of the drive. We should consider more factors to trigger gc. eg: executor exit code, task exception, task gc time .
[~pwendell]'s proposal:
if we detect memory pressure on the executors we should try to trigger a GC on the driver so that if there happen to be RDD's that have gone out of scope on the driver side, their associated cache blocks will be cleaned up on executors and free up memory.
was:
[SPARK-1103|https://issues.apache.org/jira/browse/SPARK-1103] implementation GC-based cleaning only consider the memory usage of the drive. We should consider more factors to trigger gc.includes executor exit code, task exception, task gc time .
[~pwendell]'s proposal:
if we detect memory pressure on the executors we should try to trigger a GC on the driver so that if there happen to be RDD's that have gone out of scope on the driver side, their associated cache blocks will be cleaned up on executors and free up memory.
> The driver run garbage collection, when the executor throws OutOfMemoryError exception
> --------------------------------------------------------------------------------------
>
> Key: SPARK-2595
> URL: https://issues.apache.org/jira/browse/SPARK-2595
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Guoqiang Li
>
> [SPARK-1103|https://issues.apache.org/jira/browse/SPARK-1103] implementation GC-based cleaning only consider the memory usage of the drive. We should consider more factors to trigger gc. eg: executor exit code, task exception, task gc time .
> [~pwendell]'s proposal:
> if we detect memory pressure on the executors we should try to trigger a GC on the driver so that if there happen to be RDD's that have gone out of scope on the driver side, their associated cache blocks will be cleaned up on executors and free up memory.
--
This message was sent by Atlassian JIRA
(v6.2#6252)