You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "SongYadong (JIRA)" <ji...@apache.org> on 2019/02/20 07:21:01 UTC
[jira] [Resolved] (SPARK-26884) Let task acquire memory accurately
when using spilled memory
[ https://issues.apache.org/jira/browse/SPARK-26884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
SongYadong resolved SPARK-26884.
--------------------------------
Resolution: Not A Problem
Before spill, we have called `acquireExecutionMemory `in which task already got at least `1 / (2 * numActiveTasks)` of total memory pool. But `numActiveTasks `may have changed when calling `acquireExecutionMemory `again after spill. That's my mistake.
If task havn't got `1 / (2 * numActiveTasks)` of total memory pool, the change proposed in this issue will in some cases result in more spill instead of waiting for other tasks to free up more memory. This result may lead to direction we don't want. And spill is more expensive generally.
So this is not a problem.
> Let task acquire memory accurately when using spilled memory
> ------------------------------------------------------------
>
> Key: SPARK-26884
> URL: https://issues.apache.org/jira/browse/SPARK-26884
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: SongYadong
> Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> When task can't get required execution memory, it will call *spill()* of consumers to release more memory. After spilling, it tries again to acquire memory needed which is *(required - got)*. That's not accurate as actually *released* memory by spilling may not equal to size it try to spill.
> So it may be better to acquire memory in more accurate size :
> 1. when *released* >= needed, acquire needed size. (*required - got*)
> 2. when *released* < needed, acquire released size.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org