You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2014/11/14 20:46:35 UTC

[jira] [Commented] (SPARK-4410) Support for external sort

    [ https://issues.apache.org/jira/browse/SPARK-4410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212729#comment-14212729 ] 

Apache Spark commented on SPARK-4410:
-------------------------------------

User 'marmbrus' has created a pull request for this issue:
https://github.com/apache/spark/pull/3268

> Support for external sort
> -------------------------
>
>                 Key: SPARK-4410
>                 URL: https://issues.apache.org/jira/browse/SPARK-4410
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Michael Armbrust
>            Assignee: Michael Armbrust
>            Priority: Critical
>
> When any given key is of too high cardinality the current sorting code can tip over (since it loads a whole partition into memory).  I propose we add optional support for using sparks built in external sorting mechanism.  It can be off by default, but if we determine this code path does not regress performance we can turn it on by default in the future.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org