You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/12/22 22:09:26 UTC
spark git commit: [SPARK-4864] Add documentation to Netty-based
configs
Repository: spark
Updated Branches:
refs/heads/master 7c0ed13d2 -> fbca6b6ce
[SPARK-4864] Add documentation to Netty-based configs
Author: Aaron Davidson <aa...@databricks.com>
Closes #3713 from aarondav/netty-configs and squashes the following commits:
8a8b373 [Aaron Davidson] Address Patrick's comments
3b1f84e [Aaron Davidson] [SPARK-4864] Add documentation to Netty-based configs
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/fbca6b6c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/fbca6b6c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/fbca6b6c
Branch: refs/heads/master
Commit: fbca6b6ce293b1997b40abeb9ab77b8a969a5fc9
Parents: 7c0ed13
Author: Aaron Davidson <aa...@databricks.com>
Authored: Mon Dec 22 13:09:22 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Mon Dec 22 13:09:22 2014 -0800
----------------------------------------------------------------------
docs/configuration.md | 35 ++++++++++++++++++++
.../spark/network/util/TransportConf.java | 2 +-
2 files changed, 36 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/fbca6b6c/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index 2c8dea8..2cc013c 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -852,6 +852,41 @@ Apart from these, the following properties are also available, and may be useful
between nodes leading to flooding the network with those.
</td>
</tr>
+<tr>
+ <td><code>spark.shuffle.io.preferDirectBufs</code></td>
+ <td>true</td>
+ <td>
+ (Netty only) Off-heap buffers are used to reduce garbage collection during shuffle and cache
+ block transfer. For environments where off-heap memory is tightly limited, users may wish to
+ turn this off to force all allocations from Netty to be on-heap.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.numConnectionsPerPeer</code></td>
+ <td>1</td>
+ <td>
+ (Netty only) Connections between hosts are reused in order to reduce connection buildup for
+ large clusters. For clusters with many hard disks and few hosts, this may result in insufficient
+ concurrency to saturate all disks, and so users may consider increasing this value.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.maxRetries</code></td>
+ <td>3</td>
+ <td>
+ (Netty only) Fetches that fail due to IO-related exceptions are automatically retried if this is
+ set to a non-zero value. This retry logic helps stabilize large shuffles in the face of long GC
+ pauses or transient network connectivity issues.
+ </td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.retryWait</code></td>
+ <td>5</td>
+ <td>
+ (Netty only) Seconds to wait between retries of fetches. The maximum delay caused by retrying
+ is simply <code>maxRetries * retryWait</code>, by default 15 seconds.
+ </td>
+</tr>
</table>
#### Scheduling
http://git-wip-us.apache.org/repos/asf/spark/blob/fbca6b6c/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java
----------------------------------------------------------------------
diff --git a/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java b/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java
index 13b37f9..7c9adf5 100644
--- a/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java
+++ b/network/common/src/main/java/org/apache/spark/network/util/TransportConf.java
@@ -40,7 +40,7 @@ public class TransportConf {
return conf.getInt("spark.shuffle.io.connectionTimeout", 120) * 1000;
}
- /** Number of concurrent connections between two nodes for fetching data. **/
+ /** Number of concurrent connections between two nodes for fetching data. */
public int numConnectionsPerPeer() {
return conf.getInt("spark.shuffle.io.numConnectionsPerPeer", 1);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org