You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Liang-Chi Hsieh (JIRA)" <ji...@apache.org> on 2017/01/18 14:55:26 UTC

[jira] [Created] (SPARK-19274) Make GlobalLimit without shuffling data to single partition

Liang-Chi Hsieh created SPARK-19274:
---------------------------------------

             Summary: Make GlobalLimit without shuffling data to single partition
                 Key: SPARK-19274
                 URL: https://issues.apache.org/jira/browse/SPARK-19274
             Project: Spark
          Issue Type: Improvement
          Components: SQL
            Reporter: Liang-Chi Hsieh


A logical Limit is performed actually by two physical operations LocalLimit and GlobalLimit.

In most of time, before GlobalLimit, we will perform a shuffle exchange to shuffle data to single partition. When the limit number is not trivially small, this shuffling is costing.

This change tried to perform GlobalLimit without shuffling data to single partition. The approach is similar to SparkPlan.executeTake. It iterates part of partitions until it reaches enough data.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org