[jira] [Resolved] (SPARK-9702) Repartition operator should use Exchange to perform its shuffle

["Yin Huai (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-9702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yin Huai resolved SPARK-9702.
-----------------------------
       Resolution: Fixed
    Fix Version/s: 1.6.0

Issue resolved by pull request 8083
[https://github.com/apache/spark/pull/8083]

> Repartition operator should use Exchange to perform its shuffle
> ---------------------------------------------------------------
>
>                 Key: SPARK-9702
>                 URL: https://issues.apache.org/jira/browse/SPARK-9702
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Josh Rosen
>             Fix For: 1.6.0
>
>
> Spark SQL's {{Repartition}} operator is implemented in terms of Spark Core's repartition operator, which means that it has to perform lots of unnecessary row copying and inefficient row serialization. Instead, it would be better if this was implemented using some of Exchange's internals so that it can avoid row format conversions and generic getters / hashcodes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org