You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2015/08/06 20:24:04 UTC
[jira] [Updated] (SPARK-9693) Reserve a page in all unsafe
operators to avoid starving an operator
[ https://issues.apache.org/jira/browse/SPARK-9693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or updated SPARK-9693:
-----------------------------
Description:
E.g. Currently we can do up to 3 sorts within a task:
(1) During the aggregation
(2) During a sort on the same key
(3) During the shuffle
In environments with tight memory restrictions, the first operator may acquire so much memory such that the subsequent ones in the same task are starved. A simple fix is to reserve at least a page in advance in each of these places. The reserved page size need not be the same as the normal page size.
was:
Currently we can do up to 3 sorts within a task:
(1) During the aggregation
(2) During a sort on the same key
(3) During the shuffle
In environments with tight memory restrictions, the first operator may acquire so much memory such that the subsequent ones in the same task are starved. A simple fix is to reserve at least a page in advance in each of these places. The reserved page size need not be the same as the normal page size.
> Reserve a page in all unsafe operators to avoid starving an operator
> --------------------------------------------------------------------
>
> Key: SPARK-9693
> URL: https://issues.apache.org/jira/browse/SPARK-9693
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Critical
>
> E.g. Currently we can do up to 3 sorts within a task:
> (1) During the aggregation
> (2) During a sort on the same key
> (3) During the shuffle
> In environments with tight memory restrictions, the first operator may acquire so much memory such that the subsequent ones in the same task are starved. A simple fix is to reserve at least a page in advance in each of these places. The reserved page size need not be the same as the normal page size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org