[jira] [Commented] (SPARK-2045) Sort-based shuffle implementation

["Saisai Shao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063006#comment-14063006 ] 

Saisai Shao commented on SPARK-2045:
------------------------------------

Hi Matei, great to see your design doc, a simple question:

When will we do map-side combine in sort-based shuffle? At the beginning of SortedFileWriter, or when merging the intermediate file like MR?

Looking forward to your plan, and hopes I can share my effort to this :).


> Sort-based shuffle implementation
> ---------------------------------
>
>                 Key: SPARK-2045
>                 URL: https://issues.apache.org/jira/browse/SPARK-2045
>             Project: Spark
>          Issue Type: New Feature
>            Reporter: Matei Zaharia
>         Attachments: Sort-basedshuffledesign.pdf
>
>
> Building on the pluggability in SPARK-2044, a sort-based shuffle implementation that takes advantage of an Ordering for keys (or just sorts by hashcode for keys that don't have it) would likely improve performance and memory usage in very large shuffles. Our current hash-based shuffle needs an open file for each reduce task, which can fill up a lot of memory for compression buffers and cause inefficient IO. This would avoid both of those issues.



--
This message was sent by Atlassian JIRA
(v6.2#6252)