You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/11/01 21:15:29 UTC
git commit: [SPARK-4183] Enable NettyBlockTransferService by default
Repository: spark
Updated Branches:
refs/heads/master 7136719b7 -> 59e626c70
[SPARK-4183] Enable NettyBlockTransferService by default
Note that we're turning this on for at least the first part of the QA period as a trial. We want to enable this (and deprecate the NioBlockTransferService) as soon as possible in the hopes that NettyBlockTransferService will be more stable and easier to maintain. We will turn it off if we run into major issues.
Author: Aaron Davidson <aa...@databricks.com>
Closes #3049 from aarondav/enable-netty and squashes the following commits:
bb981cc [Aaron Davidson] [SPARK-4183] Enable NettyBlockTransferService by default
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/59e626c7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/59e626c7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/59e626c7
Branch: refs/heads/master
Commit: 59e626c701227634336110e1bc23afd94c535ede
Parents: 7136719
Author: Aaron Davidson <aa...@databricks.com>
Authored: Sat Nov 1 13:15:24 2014 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Sat Nov 1 13:15:24 2014 -0700
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/SparkEnv.scala | 2 +-
docs/configuration.md | 10 ++++++++++
2 files changed, 11 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/59e626c7/core/src/main/scala/org/apache/spark/SparkEnv.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala b/core/src/main/scala/org/apache/spark/SparkEnv.scala
index 557d2f5..16c5d66 100644
--- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
+++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
@@ -274,7 +274,7 @@ object SparkEnv extends Logging {
val shuffleMemoryManager = new ShuffleMemoryManager(conf)
val blockTransferService =
- conf.get("spark.shuffle.blockTransferService", "nio").toLowerCase match {
+ conf.get("spark.shuffle.blockTransferService", "netty").toLowerCase match {
case "netty" =>
new NettyBlockTransferService(conf)
case "nio" =>
http://git-wip-us.apache.org/repos/asf/spark/blob/59e626c7/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 3007706..78c4bf3 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -359,6 +359,16 @@ Apart from these, the following properties are also available, and may be useful
map-side aggregation and there are at most this many reduce partitions.
</td>
</tr>
+<tr>
+ <td><code>spark.shuffle.blockTransferService</code></td>
+ <td>netty</td>
+ <td>
+ Implementation to use for transferring shuffle and cached blocks between executors. There
+ are two implementations available: <code>netty</code> and <code>nio</code>. Netty-based
+ block transfer is intended to be simpler but equally efficient and is the default option
+ starting in 1.2.
+ </td>
+</tr>
</table>
#### Spark UI
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org