You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by zs...@apache.org on 2017/08/24 21:27:55 UTC
spark git commit: [SPARK-21701][CORE] Enable RPC client to use `
SO_RCVBUF` and ` SO_SNDBUF` in SparkConf.
Repository: spark
Updated Branches:
refs/heads/master d3abb3699 -> 763b83ee8
[SPARK-21701][CORE] Enable RPC client to use ` SO_RCVBUF` and ` SO_SNDBUF` in SparkConf.
## What changes were proposed in this pull request?
TCP parameters like SO_RCVBUF and SO_SNDBUF can be set in SparkConf, and `org.apache.spark.network.server.TransportServe`r can use those parameters to build server by leveraging netty. But for TransportClientFactory, there is no such way to set those parameters from SparkConf. This could be inconsistent in server and client side when people set parameters in SparkConf. So this PR make RPC client to be enable to use those TCP parameters as well.
## How was this patch tested?
Existing tests.
Author: xu.zhang <xu...@hulu.com>
Closes #18964 from neoremind/add_client_param.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/763b83ee
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/763b83ee
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/763b83ee
Branch: refs/heads/master
Commit: 763b83ee84cbb6f263218c471dd9198dd6bee411
Parents: d3abb36
Author: xu.zhang <xu...@hulu.com>
Authored: Thu Aug 24 14:27:52 2017 -0700
Committer: Shixiong Zhu <zs...@gmail.com>
Committed: Thu Aug 24 14:27:52 2017 -0700
----------------------------------------------------------------------
.../apache/spark/network/client/TransportClientFactory.java | 8 ++++++++
1 file changed, 8 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/763b83ee/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
----------------------------------------------------------------------
diff --git a/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java b/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
index b50e043..8add4e1 100644
--- a/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
+++ b/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java
@@ -210,6 +210,14 @@ public class TransportClientFactory implements Closeable {
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, conf.connectionTimeoutMs())
.option(ChannelOption.ALLOCATOR, pooledAllocator);
+ if (conf.receiveBuf() > 0) {
+ bootstrap.option(ChannelOption.SO_RCVBUF, conf.receiveBuf());
+ }
+
+ if (conf.sendBuf() > 0) {
+ bootstrap.option(ChannelOption.SO_SNDBUF, conf.sendBuf());
+ }
+
final AtomicReference<TransportClient> clientRef = new AtomicReference<>();
final AtomicReference<Channel> channelRef = new AtomicReference<>();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org