You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2017/07/27 03:55:52 UTC
spark git commit: [SPARK-21530] Update description of
spark.shuffle.maxChunksBeingTransferred.
Repository: spark
Updated Branches:
refs/heads/master 60472dbfd -> cfb25b27c
[SPARK-21530] Update description of spark.shuffle.maxChunksBeingTransferred.
## What changes were proposed in this pull request?
Update the description of `spark.shuffle.maxChunksBeingTransferred` to include that the new coming connections will be closed when the max is hit and client should have retry mechanism.
Author: jinxing <ji...@126.com>
Closes #18735 from jinxing64/SPARK-21530.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cfb25b27
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cfb25b27
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cfb25b27
Branch: refs/heads/master
Commit: cfb25b27c0b32a8a70a518955fb269314b1fd716
Parents: 60472db
Author: jinxing <ji...@126.com>
Authored: Thu Jul 27 11:55:48 2017 +0800
Committer: Wenchen Fan <we...@databricks.com>
Committed: Thu Jul 27 11:55:48 2017 +0800
----------------------------------------------------------------------
.../main/java/org/apache/spark/network/util/TransportConf.java | 6 +++++-
docs/configuration.md | 6 +++++-
2 files changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/cfb25b27/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
----------------------------------------------------------------------
diff --git a/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java b/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
index ea52e9f..88256b8 100644
--- a/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
+++ b/common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
@@ -258,7 +258,11 @@ public class TransportConf {
}
/**
- * The max number of chunks allowed to being transferred at the same time on shuffle service.
+ * The max number of chunks allowed to be transferred at the same time on shuffle service.
+ * Note that new incoming connections will be closed when the max number is hit. The client will
+ * retry according to the shuffle retry configs (see `spark.shuffle.io.maxRetries` and
+ * `spark.shuffle.io.retryWait`), if those limits are reached the task will fail with fetch
+ * failure.
*/
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
http://git-wip-us.apache.org/repos/asf/spark/blob/cfb25b27/docs/configuration.md
----------------------------------------------------------------------
diff --git a/docs/configuration.md b/docs/configuration.md
index f4b6f46..500f980 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -635,7 +635,11 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.shuffle.maxChunksBeingTransferred</code></td>
<td>Long.MAX_VALUE</td>
<td>
- The max number of chunks allowed to being transferred at the same time on shuffle service.
+ The max number of chunks allowed to be transferred at the same time on shuffle service.
+ Note that new incoming connections will be closed when the max number is hit. The client will
+ retry according to the shuffle retry configs (see <code>spark.shuffle.io.maxRetries</code> and
+ <code>spark.shuffle.io.retryWait</code>), if those limits are reached the task will fail with
+ fetch failure.
</td>
</tr>
<tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org