You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Davies Liu (JIRA)" <ji...@apache.org> on 2015/06/09 08:31:00 UTC

[jira] [Created] (SPARK-8202) PySpark: infinite loop during external sort

Davies Liu created SPARK-8202:
---------------------------------

             Summary: PySpark: infinite loop during external sort 
                 Key: SPARK-8202
                 URL: https://issues.apache.org/jira/browse/SPARK-8202
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 1.4.0
            Reporter: Davies Liu
            Assignee: Davies Liu
            Priority: Critical


The batch size during external sort will grow up to max 10000, then shrink down to zero, causing infinite loop.

Given the assumption that the items usually have similar size, so we don't need to adjust the batch size after first spill.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org