[jira] [Commented] (SPARK-1956) Enable shuffle consolidation by default

["Andrew Ash (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14124844#comment-14124844 ] 

Andrew Ash commented on SPARK-1956:
-----------------------------------

[~mridulm80] there has been a significant amount of work done on the shuffle codepath for 1.1 -- do you have more issues in mind that would block enabling shuffle consolidation by default?

{{spark.shuffle.consolidateFiles}} still defaults to false in v1.1.0-rc4

> Enable shuffle consolidation by default
> ---------------------------------------
>
>                 Key: SPARK-1956
>                 URL: https://issues.apache.org/jira/browse/SPARK-1956
>             Project: Spark
>          Issue Type: Improvement
>          Components: Shuffle, Spark Core
>    Affects Versions: 1.0.0
>            Reporter: Sandy Ryza
>
> The only drawbacks are on ext3, and most everyone has ext4 at this point.  I think it's better to aim the default at the common case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org