You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2015/08/06 20:06:05 UTC

[jira] [Created] (SPARK-9693) Reserve a page in all sorters to avoid starving an operator

Andrew Or created SPARK-9693:
--------------------------------

             Summary: Reserve a page in all sorters to avoid starving an operator
                 Key: SPARK-9693
                 URL: https://issues.apache.org/jira/browse/SPARK-9693
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 1.5.0
            Reporter: Andrew Or
            Assignee: Andrew Or
            Priority: Critical


Currently we can do up to 3 sorts within a task:

(1) During the aggregation
(2) During a sort on the same key
(3) During the shuffle

In environments with tight memory restrictions, the first operator may acquire so much memory such that the subsequent ones in the same task are starved. A simple fix is to reserve at least a page in advance in each of these places. The reserved page size need not be the same as the normal page size.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org