You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/10 08:38:12 UTC
[1/2] git commit: Enable shuffle consolidation by default.
Updated Branches:
refs/heads/master 997c830e0 -> dd03cea02
Enable shuffle consolidation by default.
Bump this to being enabled for 0.9.0.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/460f655c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/460f655c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/460f655c
Branch: refs/heads/master
Commit: 460f655cc60b1a1759b6f85ae8860595fc7586b4
Parents: 4b074fa
Author: Patrick Wendell <pw...@gmail.com>
Authored: Thu Jan 9 22:42:50 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Thu Jan 9 22:42:50 2014 -0800
----------------------------------------------------------------------
.../main/scala/org/apache/spark/storage/ShuffleBlockManager.scala | 2 +-
docs/configuration.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/460f655c/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
index e2b2429..6e0ff14 100644
--- a/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
+++ b/core/src/main/scala/org/apache/spark/storage/ShuffleBlockManager.scala
@@ -64,7 +64,7 @@ class ShuffleBlockManager(blockManager: BlockManager) {
// Turning off shuffle file consolidation causes all shuffle Blocks to get their own file.
// TODO: Remove this once the shuffle file consolidation feature is stable.
val consolidateShuffleFiles =
- conf.getBoolean("spark.shuffle.consolidateFiles", false)
+ conf.getBoolean("spark.shuffle.consolidateFiles", true)
private val bufferSize = conf.getInt("spark.shuffle.file.buffer.kb", 100) * 1024
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/460f655c/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 6717757..b1a0e19 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -371,7 +371,7 @@ Apart from these, the following properties are also available, and may be useful
<tr>
<td>spark.shuffle.consolidateFiles</td>
- <td>false</td>
+ <td>true</td>
<td>
If set to "true", consolidates intermediate files created during a shuffle. Creating fewer files can improve filesystem performance for shuffles with large numbers of reduce tasks. It is recommended to set this to "true" when using ext4 or xfs filesystems. On ext3, this option might degrade performance on machines with many (>8) cores due to filesystem limitations.
</td>
[2/2] git commit: Merge pull request #378 from pwendell/consolidate_on
Posted by pw...@apache.org.
Merge pull request #378 from pwendell/consolidate_on
Enable shuffle consolidation by default.
Bump this to being enabled for 0.9.0.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/dd03cea0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/dd03cea0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/dd03cea0
Branch: refs/heads/master
Commit: dd03cea02a55746ae7e753f00ef00fc50ecaad46
Parents: 997c830 460f655
Author: Patrick Wendell <pw...@gmail.com>
Authored: Thu Jan 9 23:38:03 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Thu Jan 9 23:38:03 2014 -0800
----------------------------------------------------------------------
.../main/scala/org/apache/spark/storage/ShuffleBlockManager.scala | 2 +-
docs/configuration.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------