You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by redsanket <gi...@git.apache.org> on 2018/08/21 17:36:27 UTC
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
GitHub user redsanket opened a pull request:
https://github.com/apache/spark/pull/22173
[SPARK-24335] Spark external shuffle server improvement to better handle block fetch requests.
## What changes were proposed in this pull request?
This is a continuation PR from https://github.com/apache/spark/pull/21402
Since there is no activity, I am willing to take this over and made few minor changes and tested them.
Adding the description from the earlier PR
Description:
Right now, the default server side netty handler threads is 2 * # cores, and can be further configured with parameter spark.shuffle.io.serverThreads.
In order to process a client request, it would require one available server netty handler thread.
However, when the server netty handler threads start to process ChunkFetchRequests, they will be blocked on disk I/O, mostly due to disk contentions from the random read operations initiated by all the ChunkFetchRequests received from clients.
As a result, when the shuffle server is serving many concurrent ChunkFetchRequests, the server side netty handler threads could all be blocked on reading shuffle files, thus leaving no handler thread available to process other types of requests which should all be very quick to process.
This issue could potentially be fixed by limiting the number of netty handler threads that could get blocked when processing ChunkFetchRequest. We have a patch to do this by using a separate EventLoopGroup with a dedicated ChannelHandler to process ChunkFetchRequest. This enables shuffle server to reserve netty handler threads for non-ChunkFetchRequest, thus enabling consistent processing time for these requests which are fast to process. After deploying the patch in our infrastructure, we no longer see timeout issues with either executor registration with local shuffle server or shuffle client establishing connection with remote shuffle server.
(Please fill in changes proposed in this fix)
## How was this patch tested?
Unit tests and stress testing.
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/redsanket/spark SPARK-24335
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22173.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22173
----
commit 44bb55759a4059d8bb0e60c361a8a3210a234f92
Author: Sanket Chintapalli <sc...@...>
Date: 2018-08-21T17:34:31Z
SPARK-24355 Spark external shuffle server improvement to better handle block fetch requests.
commit 3bab74ca84fe1b6682000741b958c8792f792472
Author: Sanket Chintapalli <sc...@...>
Date: 2018-08-21T16:49:50Z
make chunk fetch handler threads as a percentage of transport server threads
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96259 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96259/testReport)** for PR 22173 at commit [`573033c`](https://github.com/apache/spark/commit/573033c5b42abf9220b6bf656b4c2f04ea615ab7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96307 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96307/testReport)** for PR 22173 at commit [`0348ec8`](https://github.com/apache/spark/commit/0348ec8d5570aab9d744043a3d6a88950f4aeb5c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95434/testReport)** for PR 22173 at commit [`b1105bd`](https://github.com/apache/spark/commit/b1105bd159bbb8a57947d38223ca7dddf0a35299).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96208 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96208/testReport)** for PR 22173 at commit [`0a43f22`](https://github.com/apache/spark/commit/0a43f223fb97da4fcf355dba945a3a350245c5f3).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95499 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95499/testReport)** for PR 22173 at commit [`b1105bd`](https://github.com/apache/spark/commit/b1105bd159bbb8a57947d38223ca7dddf0a35299).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96307 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96307/testReport)** for PR 22173 at commit [`0348ec8`](https://github.com/apache/spark/commit/0348ec8d5570aab9d744043a3d6a88950f4aeb5c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96197/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95876 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95876/testReport)** for PR 22173 at commit [`8153de5`](https://github.com/apache/spark/commit/8153de5e4bebcad3e0faa497102898bd0bcc4483).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22173
- You can say 'Closes #21402' to automatically close the other PR if this is merged.
- Please clean up the remaining parts of the template from the PR description.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216005916
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of io.serverThreads
+ * which equals 0.1 * 2*#cores or 0.1 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if(!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
+ return this.serverThreads() > 0? (this.serverThreads() * chunkFetchHandlerThreadsPercent)/100:
+ (2* NettyRuntime.availableProcessors() * chunkFetchHandlerThreadsPercent)/100;
--- End diff --
space between 2 and *
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95423/testReport)** for PR 22173 at commit [`dcc41f5`](https://github.com/apache/spark/commit/dcc41f5f6a26878dd5c7764e432bcf10cb04d5e8).
* This patch **fails Java style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218462401
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -88,6 +97,16 @@ public TransportContext(
this.conf = conf;
this.rpcHandler = rpcHandler;
this.closeIdleConnections = closeIdleConnections;
+
+ synchronized(this.getClass()) {
--- End diff --
I think synchronized(this.getClass()) is not recommended due to it not handling inheritance and such. Use synchronized(TransportContext.class)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95499/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96305 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96305/testReport)** for PR 22173 at commit [`4c5d6f1`](https://github.com/apache/spark/commit/4c5d6f1c2e4f87e74de07435eb4bb196bd150cd0).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22173
also please fix jira number, should be SPARK-24355
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218559203
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 90 would mean 90% of 2*#cores or 90% of io.serverThreads
+ * which equals 0.9 * 2*#cores or 0.9 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if (!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
--- End diff --
Yes it is documented above... if it is 0 or 100 it is 2*#cores or io.serverThreads
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216006120
--- Diff: common/network-common/src/test/java/org/apache/spark/network/ExtendedChannelPromise.java ---
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelPromise;
+import io.netty.channel.DefaultChannelPromise;
+import io.netty.util.concurrent.Future;
+import io.netty.util.concurrent.GenericFutureListener;
+
+
--- End diff --
remove extra line.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22173
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96255 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96255/testReport)** for PR 22173 at commit [`f05924a`](https://github.com/apache/spark/commit/f05924a52f8dcb0c8acc928108628d9ddf5c33d9).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96294/testReport)** for PR 22173 at commit [`60b9347`](https://github.com/apache/spark/commit/60b93477cbe2ef00ad4bf262fb9ea246f687b8f7).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218811827
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java ---
@@ -32,7 +41,6 @@
import org.apache.spark.network.client.*;
import org.apache.spark.network.protocol.*;
import org.apache.spark.network.util.TransportFrameDecoder;
-
--- End diff --
put extra line back to keep separation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218533702
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 90 would mean 90% of 2*#cores or 90% of io.serverThreads
+ * which equals 0.9 * 2*#cores or 0.9 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if (!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
--- End diff --
what happens if this is actually 0? you are going to get 0 chunkFetchHandlerThreads so what does the event loop group do with that?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216057933
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of io.serverThreads
+ * which equals 0.1 * 2*#cores or 0.1 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if(!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
+ return this.serverThreads() > 0? (this.serverThreads() * chunkFetchHandlerThreadsPercent)/100:
--- End diff --
I assume we aren't documenting chunkFetchHandlerThreadsPercent since the serverthreads config isn't documented
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96274/testReport)** for PR 22173 at commit [`fdad272`](https://github.com/apache/spark/commit/fdad2721cc69732cc86eea5a65d995cb8e818162).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96255/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96256/testReport)** for PR 22173 at commit [`ec912d6`](https://github.com/apache/spark/commit/ec912d6ca4e6911de218dc48ea05eecf9f0ff23e).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218924588
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -173,5 +213,14 @@ private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler
conf.connectionTimeoutMs(), closeIdleConnections);
}
+ /**
+ * Creates the dedicated ChannelHandler for ChunkFetchRequest messages.
+ */
+ private ChunkFetchRequestHandler createChunkFetchHandler(TransportChannelHandler channelHandler,
+ RpcHandler rpcHandler) {
+ return new ChunkFetchRequestHandler(channelHandler.getClient(),
+ rpcHandler.getStreamManager(), conf.maxChunksBeingTransferred());
--- End diff --
identation 2 spaces inside return
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95289 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95289/testReport)** for PR 22173 at commit [`cc40d9b`](https://github.com/apache/spark/commit/cc40d9beb518d961f021835228d9ada60f1388cf).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96274/testReport)** for PR 22173 at commit [`fdad272`](https://github.com/apache/spark/commit/fdad2721cc69732cc86eea5a65d995cb8e818162).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96140 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96140/testReport)** for PR 22173 at commit [`8153de5`](https://github.com/apache/spark/commit/8153de5e4bebcad3e0faa497102898bd0bcc4483).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95289 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95289/testReport)** for PR 22173 at commit [`cc40d9b`](https://github.com/apache/spark/commit/cc40d9beb518d961f021835228d9ada60f1388cf).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96293/testReport)** for PR 22173 at commit [`40cfbed`](https://github.com/apache/spark/commit/40cfbed70bd51e30ac451cb2204f34c7105fa15f).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96202/testReport)** for PR 22173 at commit [`9bf5fe0`](https://github.com/apache/spark/commit/9bf5fe09c0b21e78357ef5ef7b41063c6fd7ef88).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218811115
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()), cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx, final ChunkFetchRequest msg)
+ throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel, new ChunkFetchFailure(msg.streamChunkId,
+ Throwables.getStackTraceAsString(e)));
+ return;
+ }
+
+ streamManager.chunkBeingSent(msg.streamChunkId.streamId);
+ respond(channel, new ChunkFetchSuccess(msg.streamChunkId, buf)).addListener(
+ (ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
+ }
+
+ /**
+ * The invocation to channel.writeAndFlush is async, and the actual I/O on the
+ * channel will be handled by the EventLoop the channel is registered to. So even
+ * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
+ * which is the potentially blocking call that could deplete server handler threads, is still
+ * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
+ * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
+ * calling channel.writeAndFlush will wait for the completion of sending response back to
+ * client by invoking await(). This will throttle the rate at which threads from
+ * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
+ * default EventLoopGroup, thus making sure that we can reserve some threads in
+ * TransportServer's default EventLoopGroup for handling other RPC messages.
+ */
+ private ChannelFuture respond(final Channel channel,
+ final Encodable result) throws InterruptedException {
--- End diff --
fix indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22173
ok to test
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
@tgravescs @vanzin @Victsm please chime in thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218465817
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -144,14 +163,21 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
- channel.pipeline()
+ ChunkFetchRequestHandler chunkFetchHandler =
+ createChunkFetchHandler(channelHandler, channelRpcHandler);
--- End diff --
fix spacing, only indented 2 spaces
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216005137
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of io.serverThreads
+ * which equals 0.1 * 2*#cores or 0.1 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if(!this.getModuleName().equalsIgnoreCase("shuffle")) {
--- End diff --
space after if before (
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96281/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96273/testReport)** for PR 22173 at commit [`1ea0655`](https://github.com/apache/spark/commit/1ea0655c9001fb8cd257ee5d677d4324658dfe28).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218926387
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -77,17 +82,43 @@
private static final MessageEncoder ENCODER = MessageEncoder.INSTANCE;
private static final MessageDecoder DECODER = MessageDecoder.INSTANCE;
+ // Separate thread pool for handling ChunkFetchRequest. This helps to enable throttling
+ // max number of TransportServer worker threads that are blocked on writing response
+ // of ChunkFetchRequest message back to the client via the underlying channel.
+ private static EventLoopGroup chunkFetchWorkers;
+
public TransportContext(TransportConf conf, RpcHandler rpcHandler) {
- this(conf, rpcHandler, false);
+ this(conf, rpcHandler, false, false);
}
public TransportContext(
TransportConf conf,
RpcHandler rpcHandler,
boolean closeIdleConnections) {
+ this(conf, rpcHandler, closeIdleConnections, false);
+ }
+
+ public TransportContext(
+ TransportConf conf,
+ RpcHandler rpcHandler,
+ boolean closeIdleConnections,
+ boolean isClient) {
--- End diff --
we should make this more clear. Because with external shuffle off, we want this on client mode as well since its really both a client and server. Perhaps we could change name to be isClientOnly, we should put some java docs on the function to describe the parameter as well.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96283 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96283/testReport)** for PR 22173 at commit [`574ba81`](https://github.com/apache/spark/commit/574ba81abf3e1d71bac84a83b80e4b67056f7442).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96208/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96294/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96293/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22173
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95423/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95876/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96197/testReport)** for PR 22173 at commit [`b488f0c`](https://github.com/apache/spark/commit/b488f0c11422488b7d68652ed4780950d1245557).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218466733
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
--- End diff --
this can actually be unwrapped here
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96256/testReport)** for PR 22173 at commit [`ec912d6`](https://github.com/apache/spark/commit/ec912d6ca4e6911de218dc48ea05eecf9f0ff23e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96291/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218468244
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel,
+ new ChunkFetchFailure(msg.streamChunkId,
--- End diff --
fix wrapping and sapcing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96283 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96283/testReport)** for PR 22173 at commit [`574ba81`](https://github.com/apache/spark/commit/574ba81abf3e1d71bac84a83b80e4b67056f7442).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24355] Spark external shuffle server impro...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r223513440
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -77,17 +82,54 @@
private static final MessageEncoder ENCODER = MessageEncoder.INSTANCE;
private static final MessageDecoder DECODER = MessageDecoder.INSTANCE;
+ // Separate thread pool for handling ChunkFetchRequest. This helps to enable throttling
+ // max number of TransportServer worker threads that are blocked on writing response
+ // of ChunkFetchRequest message back to the client via the underlying channel.
+ private static EventLoopGroup chunkFetchWorkers;
--- End diff --
Is there any special reason that this must be a global one? I have not yet looked the details. But looks like this may cause ChunkFetchIntegrationSuite flaky as there is no isolation between tests.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95042 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95042/testReport)** for PR 22173 at commit [`3bab74c`](https://github.com/apache/spark/commit/3bab74ca84fe1b6682000741b958c8792f792472).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216005883
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of io.serverThreads
--- End diff --
I realize these are just examples but normally I would expect a user to have the threads processing chunk request much more then those doing other things so perhaps the example should be like 90%.
Were any tests done to see how many threads were needed for the others, I would expect very few?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95352/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24355] Spark external shuffle server impro...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22173
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218533001
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 90 would mean 90% of 2*#cores or 90% of io.serverThreads
+ * which equals 0.9 * 2*#cores or 0.9 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if (!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
--- End diff --
fix spacing
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95299/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216004740
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of io.serverThreads
+ * which equals 0.1 * 2*#cores or 0.1 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if(!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
+ return this.serverThreads() > 0? (this.serverThreads() * chunkFetchHandlerThreadsPercent)/100:
--- End diff --
space between 0 and ?, and space between 100 and :
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95289/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96208 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96208/testReport)** for PR 22173 at commit [`0a43f22`](https://github.com/apache/spark/commit/0a43f223fb97da4fcf355dba945a3a350245c5f3).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96294/testReport)** for PR 22173 at commit [`60b9347`](https://github.com/apache/spark/commit/60b93477cbe2ef00ad4bf262fb9ea246f687b8f7).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216074852
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel,
+ new ChunkFetchFailure(msg.streamChunkId,
+ Throwables.getStackTraceAsString(e)));
+ return;
+ }
+
+ streamManager.chunkBeingSent(msg.streamChunkId.streamId);
+ respond(channel, new ChunkFetchSuccess(msg.streamChunkId, buf)).addListener(
+ (ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
+ }
+
+ /**
+ * The invocation to channel.writeAndFlush is async, and the actual I/O on the
+ * channel will be handled by the EventLoop the channel is registered to. So even
+ * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
+ * which is the potentially blocking call that could deplete server handler threads, is still
+ * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
+ * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
+ * calling channel.writeAndFlush will wait for the completion of sending response back to
+ * client by invoking sync(). This will throttle the rate at which threads from
+ * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
+ * default EventLoopGroup, thus making sure that we can reserve some threads in
+ * TransportServer's default EventLoopGroup for handling other RPC messages.
+ */
+ private ChannelFuture respond(final Channel channel,
+ final Encodable result) throws InterruptedException {
+ final SocketAddress remoteAddress = channel.remoteAddress();
+ return channel.writeAndFlush(result).sync().addListener((ChannelFutureListener) future -> {
--- End diff --
Yes await can be used as well... i will test it out and let you know its ramifications if any, thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218163618
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel,
+ new ChunkFetchFailure(msg.streamChunkId,
+ Throwables.getStackTraceAsString(e)));
+ return;
+ }
+
+ streamManager.chunkBeingSent(msg.streamChunkId.streamId);
+ respond(channel, new ChunkFetchSuccess(msg.streamChunkId, buf)).addListener(
+ (ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
+ }
+
+ /**
+ * The invocation to channel.writeAndFlush is async, and the actual I/O on the
+ * channel will be handled by the EventLoop the channel is registered to. So even
+ * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
+ * which is the potentially blocking call that could deplete server handler threads, is still
+ * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
+ * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
+ * calling channel.writeAndFlush will wait for the completion of sending response back to
+ * client by invoking sync(). This will throttle the rate at which threads from
+ * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
+ * default EventLoopGroup, thus making sure that we can reserve some threads in
+ * TransportServer's default EventLoopGroup for handling other RPC messages.
+ */
+ private ChannelFuture respond(final Channel channel,
+ final Encodable result) throws InterruptedException {
+ final SocketAddress remoteAddress = channel.remoteAddress();
+ return channel.writeAndFlush(result).sync().addListener((ChannelFutureListener) future -> {
--- End diff --
ok i figured the following... i think it better to rethrow a.k.a. use sync() instead of quitely logging the exception. The reason being as follows....
The respond call made here https://github.com/apache/spark/pull/22173/files#diff-a37b07d454be4d6cd26edb294661d4e3R106 waits for sending the failed or success response back. Hence, it is a blocking call. If an exception is thrown using await() we log quitely and do no rethrow, the underlying rpc handler will not know about the request status here https://github.com/apache/spark/pull/22173/files#diff-a37b07d454be4d6cd26edb294661d4e3R106, rethrowing the exception allows it to handle and throw an rpc failure here https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java#L194 else we have to wait for a timeout exception instead of an Interrupt exception. Either cases this seems fine but using sync() we will fail fast...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95300 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95300/testReport)** for PR 22173 at commit [`d86503c`](https://github.com/apache/spark/commit/d86503cf34f66d7082df8677e78f5f793e1064a0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96259 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96259/testReport)** for PR 22173 at commit [`573033c`](https://github.com/apache/spark/commit/573033c5b42abf9220b6bf656b4c2f04ea615ab7).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96264/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22173
merged into master (2.5.0)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96140 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96140/testReport)** for PR 22173 at commit [`8153de5`](https://github.com/apache/spark/commit/8153de5e4bebcad3e0faa497102898bd0bcc4483).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96255 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96255/testReport)** for PR 22173 at commit [`f05924a`](https://github.com/apache/spark/commit/f05924a52f8dcb0c8acc928108628d9ddf5c33d9).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218939333
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -98,21 +98,32 @@ public TransportContext(
this(conf, rpcHandler, closeIdleConnections, false);
}
+ /**
+ *
+ * @param conf TransportConf
+ * @param rpcHandler RpcHandler responsible for handling requests and responses.
+ * @param closeIdleConnections Close idle connections if is set to true.
+ * @param isClientOnly This config is more important when external shuffle is enabled.
--- End diff --
I think for comments we follow the same spacing convention as observed here so sticking with it...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96281/testReport)** for PR 22173 at commit [`5d4b477`](https://github.com/apache/spark/commit/5d4b477595d3c2df5722ae5cb042a16c2f53fc9b).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22173
@Victsm let me know if you are reviewing.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95434/testReport)** for PR 22173 at commit [`b1105bd`](https://github.com/apache/spark/commit/b1105bd159bbb8a57947d38223ca7dddf0a35299).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24355] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r227118510
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -77,17 +82,54 @@
private static final MessageEncoder ENCODER = MessageEncoder.INSTANCE;
private static final MessageDecoder DECODER = MessageDecoder.INSTANCE;
+ // Separate thread pool for handling ChunkFetchRequest. This helps to enable throttling
+ // max number of TransportServer worker threads that are blocked on writing response
+ // of ChunkFetchRequest message back to the client via the underlying channel.
+ private static EventLoopGroup chunkFetchWorkers;
--- End diff --
I haven't been able to reproduce this but the number of threads used for this tests are 2* number of cores or spark.shuffle.io.serverThreads.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96281/testReport)** for PR 22173 at commit [`5d4b477`](https://github.com/apache/spark/commit/5d4b477595d3c2df5722ae5cb042a16c2f53fc9b).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218562181
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -88,6 +97,16 @@ public TransportContext(
this.conf = conf;
this.rpcHandler = rpcHandler;
this.closeIdleConnections = closeIdleConnections;
+
+ synchronized(this.getClass()) {
+ if (chunkFetchWorkers == null && conf.getModuleName() != null &&
+ conf.getModuleName().equalsIgnoreCase("shuffle")) {
+ chunkFetchWorkers = NettyUtils.createEventLoop(
+ IOMode.valueOf(conf.ioMode()),
+ conf.chunkFetchHandlerThreads(),
+ "chunk-fetch-handler");
--- End diff --
like you mention if we can not create the event loop when on the client side that would be best
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95434/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95300/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96202/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96305 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96305/testReport)** for PR 22173 at commit [`4c5d6f1`](https://github.com/apache/spark/commit/4c5d6f1c2e4f87e74de07435eb4bb196bd150cd0).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95293/testReport)** for PR 22173 at commit [`6580ff1`](https://github.com/apache/spark/commit/6580ff1abec42f640c3090edfa32466f8f5b5212).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96307/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218812503
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +284,44 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Check if it is a shuffle client
+ * and avoid creating unnecessary event loops
+ * in the TransportChannelHandler
+ */
+ public boolean shuffleClient() {
+ return this.isShuffleClient;
+ }
+
+ public void setShuffleClient(boolean isShuffleClient) {
+ this.isShuffleClient = isShuffleClient;
+ }
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 90 would mean 90% of 2*#cores or 90% of io.serverThreads
+ * which equals 0.9 * 2*#cores or 0.9 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if (!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
--- End diff --
fix indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96273/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95042/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96291 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96291/testReport)** for PR 22173 at commit [`574ba81`](https://github.com/apache/spark/commit/574ba81abf3e1d71bac84a83b80e4b67056f7442).
* This patch **fails Java style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96293/testReport)** for PR 22173 at commit [`40cfbed`](https://github.com/apache/spark/commit/40cfbed70bd51e30ac451cb2204f34c7105fa15f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96273/testReport)** for PR 22173 at commit [`1ea0655`](https://github.com/apache/spark/commit/1ea0655c9001fb8cd257ee5d677d4324658dfe28).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218621553
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel,
+ new ChunkFetchFailure(msg.streamChunkId,
+ Throwables.getStackTraceAsString(e)));
+ return;
+ }
+
+ streamManager.chunkBeingSent(msg.streamChunkId.streamId);
+ respond(channel, new ChunkFetchSuccess(msg.streamChunkId, buf)).addListener(
+ (ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
+ }
+
+ /**
+ * The invocation to channel.writeAndFlush is async, and the actual I/O on the
+ * channel will be handled by the EventLoop the channel is registered to. So even
+ * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
+ * which is the potentially blocking call that could deplete server handler threads, is still
+ * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
+ * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
+ * calling channel.writeAndFlush will wait for the completion of sending response back to
+ * client by invoking sync(). This will throttle the rate at which threads from
+ * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
+ * default EventLoopGroup, thus making sure that we can reserve some threads in
+ * TransportServer's default EventLoopGroup for handling other RPC messages.
+ */
+ private ChannelFuture respond(final Channel channel,
+ final Encodable result) throws InterruptedException {
+ final SocketAddress remoteAddress = channel.remoteAddress();
+ return channel.writeAndFlush(result).sync().addListener((ChannelFutureListener) future -> {
--- End diff --
It justs logs the exception but the behaviour is same… as in the first case an additional warning from netty side is passed onto the listener and the channel is closed… in the later we quitely log error and the channel is closed… the executors deal with the closed channel and retry or schedule the task on different executor… and eventually the job seems to succeed… in the example i ran… i expect the retry to go through in such scenarios… i will just use await() as the ERROR is logged and we dont have to be more verbose with the warning...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95300 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95300/testReport)** for PR 22173 at commit [`d86503c`](https://github.com/apache/spark/commit/d86503cf34f66d7082df8677e78f5f793e1064a0).
* This patch **fails Java style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95352 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95352/testReport)** for PR 22173 at commit [`470e9a6`](https://github.com/apache/spark/commit/470e9a60c8708ebe5fc819ba95481a59b6dcfc82).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96256/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95876 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95876/testReport)** for PR 22173 at commit [`8153de5`](https://github.com/apache/spark/commit/8153de5e4bebcad3e0faa497102898bd0bcc4483).
* This patch **fails from timeout after a configured wait of \`400m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218930135
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -77,17 +82,43 @@
private static final MessageEncoder ENCODER = MessageEncoder.INSTANCE;
private static final MessageDecoder DECODER = MessageDecoder.INSTANCE;
+ // Separate thread pool for handling ChunkFetchRequest. This helps to enable throttling
+ // max number of TransportServer worker threads that are blocked on writing response
+ // of ChunkFetchRequest message back to the client via the underlying channel.
+ private static EventLoopGroup chunkFetchWorkers;
+
public TransportContext(TransportConf conf, RpcHandler rpcHandler) {
- this(conf, rpcHandler, false);
+ this(conf, rpcHandler, false, false);
}
public TransportContext(
TransportConf conf,
RpcHandler rpcHandler,
boolean closeIdleConnections) {
+ this(conf, rpcHandler, closeIdleConnections, false);
+ }
+
+ public TransportContext(
+ TransportConf conf,
+ RpcHandler rpcHandler,
+ boolean closeIdleConnections,
+ boolean isClient) {
--- End diff --
sure... anything to make is more clear
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218874415
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +284,44 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Check if it is a shuffle client
+ * and avoid creating unnecessary event loops
+ * in the TransportChannelHandler
+ */
+ public boolean shuffleClient() {
--- End diff --
I think since this is so specialized and really only used from one place (the external shuffle service client) perhaps putting it here isn't that useful. I think if we add a new constructor to the TransportContext with that optional parameter it might make more sense
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218465153
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -88,6 +97,16 @@ public TransportContext(
this.conf = conf;
this.rpcHandler = rpcHandler;
this.closeIdleConnections = closeIdleConnections;
+
+ synchronized(this.getClass()) {
+ if (chunkFetchWorkers == null && conf.getModuleName() != null &&
+ conf.getModuleName().equalsIgnoreCase("shuffle")) {
+ chunkFetchWorkers = NettyUtils.createEventLoop(
+ IOMode.valueOf(conf.ioMode()),
+ conf.chunkFetchHandlerThreads(),
+ "chunk-fetch-handler");
--- End diff --
for consistency perhaps name thread shuffle-chunk-fetch-handler
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218467664
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
--- End diff --
fix wrapping
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218590278
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel,
+ new ChunkFetchFailure(msg.streamChunkId,
+ Throwables.getStackTraceAsString(e)));
+ return;
+ }
+
+ streamManager.chunkBeingSent(msg.streamChunkId.streamId);
+ respond(channel, new ChunkFetchSuccess(msg.streamChunkId, buf)).addListener(
+ (ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
+ }
+
+ /**
+ * The invocation to channel.writeAndFlush is async, and the actual I/O on the
+ * channel will be handled by the EventLoop the channel is registered to. So even
+ * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
+ * which is the potentially blocking call that could deplete server handler threads, is still
+ * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
+ * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
+ * calling channel.writeAndFlush will wait for the completion of sending response back to
+ * client by invoking sync(). This will throttle the rate at which threads from
+ * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
+ * default EventLoopGroup, thus making sure that we can reserve some threads in
+ * TransportServer's default EventLoopGroup for handling other RPC messages.
+ */
+ private ChannelFuture respond(final Channel channel,
+ final Encodable result) throws InterruptedException {
+ final SocketAddress remoteAddress = channel.remoteAddress();
+ return channel.writeAndFlush(result).sync().addListener((ChannelFutureListener) future -> {
--- End diff --
ok i had chunkFetchHandler as an instance of rpcHandler in my mind...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r212639516
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -144,14 +161,17 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
+ ChunkFetchRequestHandler chunkFetchHandler = createChunkFetchHandler(channelHandler, channelRpcHandler);
channel.pipeline()
.addLast("encoder", ENCODER)
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", DECODER)
.addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
// NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
// would require more logic to guarantee if this were not part of the same event loop.
- .addLast("handler", channelHandler);
+ .addLast("handler", channelHandler)
+ // Use a separate EventLoopGroup to handle ChunkFetchRequest messages.
+ .addLast(chunkFetchWorkers, "chunkFetchHandler", chunkFetchHandler);
--- End diff --
yes i did notice that... makes sense
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95293/testReport)** for PR 22173 at commit [`6580ff1`](https://github.com/apache/spark/commit/6580ff1abec42f640c3090edfa32466f8f5b5212).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96274/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96283/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22173
add reference to original PR at https://github.com/apache/spark/pull/21402
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218508374
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel,
+ new ChunkFetchFailure(msg.streamChunkId,
+ Throwables.getStackTraceAsString(e)));
+ return;
+ }
+
+ streamManager.chunkBeingSent(msg.streamChunkId.streamId);
+ respond(channel, new ChunkFetchSuccess(msg.streamChunkId, buf)).addListener(
+ (ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
+ }
+
+ /**
+ * The invocation to channel.writeAndFlush is async, and the actual I/O on the
+ * channel will be handled by the EventLoop the channel is registered to. So even
+ * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
+ * which is the potentially blocking call that could deplete server handler threads, is still
+ * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
+ * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
+ * calling channel.writeAndFlush will wait for the completion of sending response back to
+ * client by invoking sync(). This will throttle the rate at which threads from
+ * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
+ * default EventLoopGroup, thus making sure that we can reserve some threads in
+ * TransportServer's default EventLoopGroup for handling other RPC messages.
+ */
+ private ChannelFuture respond(final Channel channel,
+ final Encodable result) throws InterruptedException {
+ final SocketAddress remoteAddress = channel.remoteAddress();
+ return channel.writeAndFlush(result).sync().addListener((ChannelFutureListener) future -> {
--- End diff --
so it doesn't go through https://github.com/apache/spark/blob/master/common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java#L183 since that is hte rpchandler . The TransportChannelHandler won't handle it at all, it will go to the pipeline which calls the ChunkFetchRequestHandler. So whatever is calling that is what would get the exception propogated. The reason I said this is because previously when this was handled in TransportChannelHandler, it wasn't throwing the exception up. The TrasnportRequestHandler.respond is calling hte writeAndFlush async and then just adding a listener which would simply log an error if the future wasn't successful.
I want to make sure if an exception is thrown here it doesn't kill the entire external shuffle service for instance.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218942035
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -77,17 +82,54 @@
private static final MessageEncoder ENCODER = MessageEncoder.INSTANCE;
private static final MessageDecoder DECODER = MessageDecoder.INSTANCE;
+ // Separate thread pool for handling ChunkFetchRequest. This helps to enable throttling
+ // max number of TransportServer worker threads that are blocked on writing response
+ // of ChunkFetchRequest message back to the client via the underlying channel.
+ private static EventLoopGroup chunkFetchWorkers;
+
public TransportContext(TransportConf conf, RpcHandler rpcHandler) {
- this(conf, rpcHandler, false);
+ this(conf, rpcHandler, false, false);
}
public TransportContext(
TransportConf conf,
RpcHandler rpcHandler,
boolean closeIdleConnections) {
+ this(conf, rpcHandler, closeIdleConnections, false);
+ }
+
+ /**
+ *
+ * @param conf TransportConf
+ * @param rpcHandler RpcHandler responsible for handling requests and responses.
+ * @param closeIdleConnections Close idle connections if it is set to true.
+ * @param isClientOnly This config is more important when external shuffle is enabled.
+ * It stops creating extra event loop and subsequent thread pool
+ * for shuffle clients to handle chunked fetch requests.
+ * In the case when external shuffle is disabled, the executors are both
+ * client and server so both share the same event loop which is trivial.
--- End diff --
I hope we follow a similar indentation for all other comments
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96264/testReport)** for PR 22173 at commit [`e7b47e9`](https://github.com/apache/spark/commit/e7b47e9c37e42e8de251f9f91d9f85428ea7df73).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95352 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95352/testReport)** for PR 22173 at commit [`470e9a6`](https://github.com/apache/spark/commit/470e9a60c8708ebe5fc819ba95481a59b6dcfc82).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218924768
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
--- End diff --
remove extra empty line
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218812564
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +284,44 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Check if it is a shuffle client
+ * and avoid creating unnecessary event loops
+ * in the TransportChannelHandler
+ */
+ public boolean shuffleClient() {
+ return this.isShuffleClient;
+ }
+
+ public void setShuffleClient(boolean isShuffleClient) {
+ this.isShuffleClient = isShuffleClient;
+ }
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 90 would mean 90% of 2*#cores or 90% of io.serverThreads
+ * which equals 0.9 * 2*#cores or 0.9 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if (!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
+ return this.serverThreads() > 0 ? (this.serverThreads() * chunkFetchHandlerThreadsPercent)/100:
+ (2 * NettyRuntime.availableProcessors() * chunkFetchHandlerThreadsPercent)/100;
--- End diff --
fix indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96197/testReport)** for PR 22173 at commit [`b488f0c`](https://github.com/apache/spark/commit/b488f0c11422488b7d68652ed4780950d1245557).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218807868
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()), cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx, final ChunkFetchRequest msg)
+ throws Exception {
--- End diff --
indent this 2 more spaces
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96259/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95299 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95299/testReport)** for PR 22173 at commit [`50258f7`](https://github.com/apache/spark/commit/50258f7595a49373d64d8831ff3ce410eef6e0cf).
* This patch **fails Java style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95299 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95299/testReport)** for PR 22173 at commit [`50258f7`](https://github.com/apache/spark/commit/50258f7595a49373d64d8831ff3ce410eef6e0cf).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216068689
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of io.serverThreads
--- End diff --
No based on how many threads required for other rpc calls, i have not tested them, but the whole point would be to reduce the dependency how much time the chunkFetchedRequests will be spending doing disk I/O
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95499 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95499/testReport)** for PR 22173 at commit [`b1105bd`](https://github.com/apache/spark/commit/b1105bd159bbb8a57947d38223ca7dddf0a35299).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218811721
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java ---
@@ -24,6 +24,15 @@
import com.google.common.base.Throwables;
import io.netty.channel.Channel;
import io.netty.channel.ChannelFuture;
+
+import org.apache.spark.network.protocol.OneWayMessage;
+import org.apache.spark.network.protocol.RpcFailure;
+import org.apache.spark.network.protocol.RpcRequest;
--- End diff --
these don't seem like should be necessary? and in wrong location
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216063759
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
+ ctx.close();
+ }
+
+ @Override
+ protected void channelRead0(ChannelHandlerContext ctx,
+ final ChunkFetchRequest msg) throws Exception {
+ Channel channel = ctx.channel();
+ if (logger.isTraceEnabled()) {
+ logger.trace("Received req from {} to fetch block {}", getRemoteAddress(channel),
+ msg.streamChunkId);
+ }
+ long chunksBeingTransferred = streamManager.chunksBeingTransferred();
+ if (chunksBeingTransferred >= maxChunksBeingTransferred) {
+ logger.warn("The number of chunks being transferred {} is above {}, close the connection.",
+ chunksBeingTransferred, maxChunksBeingTransferred);
+ channel.close();
+ return;
+ }
+ ManagedBuffer buf;
+ try {
+ streamManager.checkAuthorization(client, msg.streamChunkId.streamId);
+ streamManager.registerChannel(channel, msg.streamChunkId.streamId);
+ buf = streamManager.getChunk(msg.streamChunkId.streamId, msg.streamChunkId.chunkIndex);
+ } catch (Exception e) {
+ logger.error(String.format("Error opening block %s for request from %s",
+ msg.streamChunkId, getRemoteAddress(channel)), e);
+ respond(channel,
+ new ChunkFetchFailure(msg.streamChunkId,
+ Throwables.getStackTraceAsString(e)));
+ return;
+ }
+
+ streamManager.chunkBeingSent(msg.streamChunkId.streamId);
+ respond(channel, new ChunkFetchSuccess(msg.streamChunkId, buf)).addListener(
+ (ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
+ }
+
+ /**
+ * The invocation to channel.writeAndFlush is async, and the actual I/O on the
+ * channel will be handled by the EventLoop the channel is registered to. So even
+ * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
+ * which is the potentially blocking call that could deplete server handler threads, is still
+ * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
+ * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
+ * calling channel.writeAndFlush will wait for the completion of sending response back to
+ * client by invoking sync(). This will throttle the rate at which threads from
+ * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
+ * default EventLoopGroup, thus making sure that we can reserve some threads in
+ * TransportServer's default EventLoopGroup for handling other RPC messages.
+ */
+ private ChannelFuture respond(final Channel channel,
+ final Encodable result) throws InterruptedException {
+ final SocketAddress remoteAddress = channel.remoteAddress();
+ return channel.writeAndFlush(result).sync().addListener((ChannelFutureListener) future -> {
--- End diff --
do we actually want to use await() here instead of sync(). Sync says "Waits for this future until it is done, and rethrows the cause of the failure if this future failed."
I'm not sure we want a rethrow here since we have the addListener to handle the future failure.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218925291
--- Diff: common/network-common/src/test/java/org/apache/spark/network/ChunkFetchRequestHandlerSuite.java ---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network;
+
+import io.netty.channel.ChannelHandlerContext;
+import java.util.ArrayList;
+import java.util.List;
+
+import io.netty.channel.Channel;
+import org.apache.spark.network.server.ChunkFetchRequestHandler;
+import org.junit.Test;
+
+import static org.mockito.Mockito.*;
+
+import org.apache.commons.lang3.tuple.ImmutablePair;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.*;
+import org.apache.spark.network.server.NoOpRpcHandler;
+import org.apache.spark.network.server.OneForOneStreamManager;
+import org.apache.spark.network.server.RpcHandler;
+
+public class ChunkFetchRequestHandlerSuite {
+
+ @Test
+ public void handleChunkFetchRequest() throws Exception {
+ RpcHandler rpcHandler = new NoOpRpcHandler();
+ OneForOneStreamManager streamManager = (OneForOneStreamManager) (rpcHandler.getStreamManager());
+ Channel channel = mock(Channel.class);
+ ChannelHandlerContext context = mock(ChannelHandlerContext.class);
+ when(context.channel())
+ .thenAnswer(invocationOnMock0 -> {
--- End diff --
change spacing to 2
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218466397
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java ---
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.network.server;
+
+import java.net.SocketAddress;
+
+import com.google.common.base.Throwables;
+import io.netty.channel.Channel;
+import io.netty.channel.ChannelFuture;
+import io.netty.channel.ChannelFutureListener;
+import io.netty.channel.ChannelHandlerContext;
+import io.netty.channel.SimpleChannelInboundHandler;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.spark.network.buffer.ManagedBuffer;
+import org.apache.spark.network.client.TransportClient;
+import org.apache.spark.network.protocol.ChunkFetchFailure;
+import org.apache.spark.network.protocol.ChunkFetchRequest;
+import org.apache.spark.network.protocol.ChunkFetchSuccess;
+import org.apache.spark.network.protocol.Encodable;
+
+import static org.apache.spark.network.util.NettyUtils.*;
+
+
+/**
+ * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
+ * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
+ * channel could potentially be blocked due to disk contentions. If several hundreds of clients
+ * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
+ * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
+ * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
+ * result, it would leave no threads left to process other RPC messages, which takes much less
+ * time to process, and could lead to client timing out on either performing SASL authentication,
+ * registering executors, or waiting for response for an OpenBlocks messages.
+ */
+public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+ private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
+
+ private final TransportClient client;
+ private final StreamManager streamManager;
+ /** The max number of chunks being transferred and not finished yet. */
+ private final long maxChunksBeingTransferred;
+
+ public ChunkFetchRequestHandler(
+ TransportClient client,
+ StreamManager streamManager,
+ Long maxChunksBeingTransferred) {
+ this.client = client;
+ this.streamManager = streamManager;
+ this.maxChunksBeingTransferred = maxChunksBeingTransferred;
+ }
+
+ @Override
+ public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
+ logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()),
+ cause);
--- End diff --
spacing 2, fix throughout the file
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95423/testReport)** for PR 22173 at commit [`dcc41f5`](https://github.com/apache/spark/commit/dcc41f5f6a26878dd5c7764e432bcf10cb04d5e8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/22173
test this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r216069578
--- Diff: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java ---
@@ -281,4 +282,31 @@ public Properties cryptoConf() {
public long maxChunksBeingTransferred() {
return conf.getLong("spark.shuffle.maxChunksBeingTransferred", Long.MAX_VALUE);
}
+
+ /**
+ * Percentage of io.serverThreads used by netty to process ChunkFetchRequest.
+ * Shuffle server will use a separate EventLoopGroup to process ChunkFetchRequest messages.
+ * Although when calling the async writeAndFlush on the underlying channel to send
+ * response back to client, the I/O on the channel is still being handled by
+ * {@link org.apache.spark.network.server.TransportServer}'s default EventLoopGroup
+ * that's registered with the Channel, by waiting inside the ChunkFetchRequest handler
+ * threads for the completion of sending back responses, we are able to put a limit on
+ * the max number of threads from TransportServer's default EventLoopGroup that are
+ * going to be consumed by writing response to ChunkFetchRequest, which are I/O intensive
+ * and could take long time to process due to disk contentions. By configuring a slightly
+ * higher number of shuffler server threads, we are able to reserve some threads for
+ * handling other RPC messages, thus making the Client less likely to experience timeout
+ * when sending RPC messages to the shuffle server. Default to 0, which is 2*#cores
+ * or io.serverThreads. 10 would mean 10% of 2*#cores or 10% of io.serverThreads
+ * which equals 0.1 * 2*#cores or 0.1 * io.serverThreads.
+ */
+ public int chunkFetchHandlerThreads() {
+ if(!this.getModuleName().equalsIgnoreCase("shuffle")) {
+ return 0;
+ }
+ int chunkFetchHandlerThreadsPercent =
+ conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0);
+ return this.serverThreads() > 0? (this.serverThreads() * chunkFetchHandlerThreadsPercent)/100:
--- End diff --
I think it is a good idea to document both as this is an important config. Let me know your thoughts
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #95042 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95042/testReport)** for PR 22173 at commit [`3bab74c`](https://github.com/apache/spark/commit/3bab74ca84fe1b6682000741b958c8792f792472).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96202/testReport)** for PR 22173 at commit [`9bf5fe0`](https://github.com/apache/spark/commit/9bf5fe09c0b21e78357ef5ef7b41063c6fd7ef88).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95293/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
thanks @vanzin, also @tgravescs gentle ping...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96305/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/22173
(I haven't forgotten about this, just haven't had the time to look at it.)
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96291 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96291/testReport)** for PR 22173 at commit [`574ba81`](https://github.com/apache/spark/commit/574ba81abf3e1d71bac84a83b80e4b67056f7442).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218490559
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -144,14 +163,21 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
- channel.pipeline()
+ ChunkFetchRequestHandler chunkFetchHandler =
+ createChunkFetchHandler(channelHandler, channelRpcHandler);
+ ChannelPipeline pipeline = channel.pipeline()
.addLast("encoder", ENCODER)
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", DECODER)
- .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
+ .addLast("idleStateHandler",
+ new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
--- End diff --
fix indentation
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218494508
--- Diff: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java ---
@@ -24,15 +24,27 @@
import com.google.common.base.Throwables;
import io.netty.channel.Channel;
import io.netty.channel.ChannelFuture;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.spark.network.buffer.ManagedBuffer;
import org.apache.spark.network.buffer.NioManagedBuffer;
-import org.apache.spark.network.client.*;
-import org.apache.spark.network.protocol.*;
+import org.apache.spark.network.client.RpcResponseCallback;
--- End diff --
revert the imports to be .*
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r212070669
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -144,14 +161,17 @@ public TransportChannelHandler initializePipeline(
RpcHandler channelRpcHandler) {
try {
TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
+ ChunkFetchRequestHandler chunkFetchHandler = createChunkFetchHandler(channelHandler, channelRpcHandler);
channel.pipeline()
.addLast("encoder", ENCODER)
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", DECODER)
.addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
// NOTE: Chunks are currently guaranteed to be returned in the order of request, but this
// would require more logic to guarantee if this were not part of the same event loop.
- .addLast("handler", channelHandler);
+ .addLast("handler", channelHandler)
+ // Use a separate EventLoopGroup to handle ChunkFetchRequest messages.
+ .addLast(chunkFetchWorkers, "chunkFetchHandler", chunkFetchHandler);
--- End diff --
Hmm... I think there is some waste here. Not all channels actually need the chunk fetch handler. Basically only the shuffle server (external or not) does. So for all other cases - RpcEnv server and clients, shuffle clients - you'd have this new thread pool just sitting there. Before you'd have the handler in the pipeline, but that's just a few bytes of memory being wasted.
It would be good to avoid that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96140/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22173: [SPARK-24335] Spark external shuffle server impro...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/22173#discussion_r218463768
--- Diff: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java ---
@@ -88,6 +97,16 @@ public TransportContext(
this.conf = conf;
this.rpcHandler = rpcHandler;
this.closeIdleConnections = closeIdleConnections;
+
+ synchronized(this.getClass()) {
+ if (chunkFetchWorkers == null && conf.getModuleName() != null &&
+ conf.getModuleName().equalsIgnoreCase("shuffle")) {
--- End diff --
fix spacing here, line up conf.getModuleName with the chunkFetchWorkers
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24335] Spark external shuffle server improvement ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22173
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22173
**[Test build #96264 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96264/testReport)** for PR 22173 at commit [`e7b47e9`](https://github.com/apache/spark/commit/e7b47e9c37e42e8de251f9f91d9f85428ea7df73).
* This patch **fails from timeout after a configured wait of `400m`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22173: [SPARK-24355] Spark external shuffle server improvement ...
Posted by redsanket <gi...@git.apache.org>.
Github user redsanket commented on the issue:
https://github.com/apache/spark/pull/22173
closes #21402
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org