You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/21 08:54:32 UTC
[GitHub] [spark] xuanyuanking opened a new pull request #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
xuanyuanking opened a new pull request #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665
### What changes were proposed in this pull request?
Fix the regression caused by #22173.
The original PR changes the logic of handling `ChunkFetchReqeust` from async to sync, that's causes the shuffle benchmark regression. This PR fixes the regression back to the async mode by adding a feature flag.
### Why are the changes needed?
Fix the shuffle performance regression described in https://github.com/apache/spark/pull/22173#issuecomment-572459561
### Does this PR introduce any user-facing change?
Add new feature flag `spark.shuffle.chunkFetchAsyncMode.enabled`
### How was this patch tested?
Existing UT.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r398328411
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
What do you mean by `the config must be set`, @xuanyuanking ? What value do you expect by default? Apparently, this is reverting SPARK-25641 without mentioning SPARK-25641. In the PR, only SPARK-24355 is mentioned.
> No need to give a default value here, when it comes to here, the config must be set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603650859
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25020/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602630721
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589561429
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602529619
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396411190
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -65,17 +52,8 @@ public ChunkFetchRequestHandler(
this.maxChunksBeingTransferred = maxChunksBeingTransferred;
}
- @Override
- public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
- logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()), cause);
- ctx.close();
- }
-
- @Override
- protected void channelRead0(
- ChannelHandlerContext ctx,
- final ChunkFetchRequest msg) throws Exception {
- Channel channel = ctx.channel();
+ public void processFetchRequest(
+ final Channel channel, final ChunkFetchRequest msg) throws Exception {
Review comment:
shall we define channel as a protected member variable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603812006
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120326/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603003685
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589638157
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118772/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603654880
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603650519
**[Test build #120310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120310/testReport)** for PR 27665 at commit [`34b52ef`](https://github.com/apache/spark/commit/34b52ef36420ecdbc9f70ee3dc36a3e72b29de1f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396856160
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java
##########
@@ -58,19 +58,22 @@
private final TransportRequestHandler requestHandler;
private final long requestTimeoutNs;
private final boolean closeIdleConnections;
+ private final boolean separateChunkFetchRequest;
Review comment:
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603650859
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25020/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603694657
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25036/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603200510
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120274/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602616587
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603811998
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589638144
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589638144
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602519280
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24903/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676958
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603003077
**[Test build #120229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120229/testReport)** for PR 27665 at commit [`da8abd3`](https://github.com/apache/spark/commit/da8abd3dad70c4430053206ea07a459439840475).
* This patch **fails PySpark pip packaging tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396410407
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -107,31 +85,10 @@ protected void channelRead0(
(ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
}
- /**
- * The invocation to channel.writeAndFlush is async, and the actual I/O on the
- * channel will be handled by the EventLoop the channel is registered to. So even
- * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
- * which is the potentially blocking call that could deplete server handler threads, is still
- * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
- * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
- * calling channel.writeAndFlush will wait for the completion of sending response back to
- * client by invoking await(). This will throttle the rate at which threads from
- * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
- * default EventLoopGroup, thus making sure that we can reserve some threads in
- * TransportServer's default EventLoopGroup for handling other RPC messages.
- */
- private ChannelFuture respond(
- final Channel channel,
- final Encodable result) throws InterruptedException {
- final SocketAddress remoteAddress = channel.remoteAddress();
- return channel.writeAndFlush(result).await().addListener((ChannelFutureListener) future -> {
- if (future.isSuccess()) {
- logger.trace("Sent result {} to client {}", result, remoteAddress);
- } else {
- logger.error(String.format("Error sending result %s to %s; closing connection",
- result, remoteAddress), future.cause());
- channel.close();
- }
- });
- }
+ public abstract ChannelFuture respond(Channel channel, Encodable result) throws Exception;
Review comment:
why we need to define this method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603654887
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25021/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603811998
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602629376
**[Test build #120196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120196/testReport)** for PR 27665 at commit [`e655227`](https://github.com/apache/spark/commit/e65522707f9e8d96f65e279628661d138ffe8f55).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#discussion_r383080838
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -124,7 +127,13 @@ private ChannelFuture respond(
final Channel channel,
final Encodable result) throws InterruptedException {
final SocketAddress remoteAddress = channel.remoteAddress();
- return channel.writeAndFlush(result).await().addListener((ChannelFutureListener) future -> {
+ ChannelFuture channelFuture = null;
+ if (chunkFetchAsyncModeEnabled) {
+ channelFuture = channel.writeAndFlush(result);
Review comment:
Thanks for the comment @otterc!
This PR aims to fix the performance regression brings by the sync mode fetching. As you mention, you find a problem with the stress test framework, I think the default mode should be the guarantee for common use cases.
For the SASL requests timing out, maybe we need more context for your internal stress test framework and analyze the root cause. IMO, there should be other configs in spark.shuffle/netty could help, not only depends on the async/sync mode here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603200496
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589561436
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23524/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r397621197
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
+ Integer.parseInt(conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent"));
int threads =
this.serverThreads() > 0 ? this.serverThreads() : 2 * NettyRuntime.availableProcessors();
return (int) Math.ceil(threads * (chunkFetchHandlerThreadsPercent / 100.0));
}
+ /**
+ * Whether to use a separate EventLoopGroup to process ChunkFetchRequest messages, it is decided
+ * by the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
+ */
+ public boolean separateChunkFetchRequest() {
+ try {
+ conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent");
Review comment:
Thanks, done in 4f42083.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676968
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120310/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396859240
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -65,17 +52,8 @@ public ChunkFetchRequestHandler(
this.maxChunksBeingTransferred = maxChunksBeingTransferred;
}
- @Override
- public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
- logger.warn("Exception in connection from " + getRemoteAddress(ctx.channel()), cause);
- ctx.close();
- }
-
- @Override
- protected void channelRead0(
- ChannelHandlerContext ctx,
- final ChunkFetchRequest msg) throws Exception {
- Channel channel = ctx.channel();
+ public void processFetchRequest(
+ final Channel channel, final ChunkFetchRequest msg) throws Exception {
Review comment:
It's OK for `TransportRequestHandler`, but for `ChunkFetchChannelHandler`, maybe get the current channel from `ChannelHandlerContext` is a safer way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396855915
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
No need to give a default value here, when it comes to here, the config must be set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602522738
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r397102478
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
+ Integer.parseInt(conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent"));
int threads =
this.serverThreads() > 0 ? this.serverThreads() : 2 * NettyRuntime.availableProcessors();
return (int) Math.ceil(threads * (chunkFetchHandlerThreadsPercent / 100.0));
}
+ /**
+ * Whether to use a separate EventLoopGroup to process ChunkFetchRequest messages, it is decided
+ * by the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
+ */
+ public boolean separateChunkFetchRequest() {
+ try {
+ conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent");
Review comment:
how about `conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 0) > 0`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589637229
**[Test build #118772 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118772/testReport)** for PR 27665 at commit [`743e566`](https://github.com/apache/spark/commit/743e566f952648be43fca1cc6e6781030651c1ce).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396404060
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
+ Integer.parseInt(conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent"));
int threads =
this.serverThreads() > 0 ? this.serverThreads() : 2 * NettyRuntime.availableProcessors();
return (int) Math.ceil(threads * (chunkFetchHandlerThreadsPercent / 100.0));
}
+ /**
+ * Whether to use a separate EventLoopGroup to process ChunkFetchRequest messages, it is decided
+ * by the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
+ */
+ public boolean separateChunkFetchRequest() {
+ try {
+ conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent");
Review comment:
there is no API to check conf existence?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603202843
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602529619
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676767
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-604227335
Hi, @cloud-fan . This seems to be not in `branch-3.0` yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603200461
**[Test build #120274 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120274/testReport)** for PR 27665 at commit [`34b52ef`](https://github.com/apache/spark/commit/34b52ef36420ecdbc9f70ee3dc36a3e72b29de1f).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602961327
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603694657
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25036/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603654493
**[Test build #120311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120311/testReport)** for PR 27665 at commit [`4f42083`](https://github.com/apache/spark/commit/4f42083b43058d8f7276e5f095dc4b432c8b2b71).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602519265
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603200496
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603654493
**[Test build #120311 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120311/testReport)** for PR 27665 at commit [`4f42083`](https://github.com/apache/spark/commit/4f42083b43058d8f7276e5f095dc4b432c8b2b71).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602519265
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r398328411
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
What do you mean by `the config must be set`, @xuanyuanking ? What value do you expect by default? Apparently, this seems to revert SPARK-25641 together without mentioning SPARK-25641. In the PR, only SPARK-24355 is mentioned.
> No need to give a default value here, when it comes to here, the config must be set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-590816607
The default value of `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is 100, so this feature is expected to be disabled by default IIUC.
Policy-wise, we should not fix an issue while introducing a regression. I'm OK to have it since it's disabled by default and it does fix a common issue. What I'm asking for is to make sure the code path is exactly the same as before when this feature is disabled, so that there is no regression.
+1 with @tgravescs to reuse the existing config.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396859444
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java
##########
@@ -67,6 +68,7 @@ public TransportRequestHandler(
TransportClient reverseClient,
RpcHandler rpcHandler,
Long maxChunksBeingTransferred) {
+ super(reverseClient, rpcHandler.getStreamManager(), maxChunksBeingTransferred);
Review comment:
Yes, it is still in use in `processStreamRequest`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603202850
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120273/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603812006
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120326/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603189973
**[Test build #120273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120273/testReport)** for PR 27665 at commit [`d4e0352`](https://github.com/apache/spark/commit/d4e0352ba54f5e05aae93b1e19183988e2d9f7e6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602630721
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603810326
**[Test build #120326 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120326/testReport)** for PR 27665 at commit [`4f42083`](https://github.com/apache/spark/commit/4f42083b43058d8f7276e5f095dc4b432c8b2b71).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602616587
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602522081
**[Test build #120192 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120192/testReport)** for PR 27665 at commit [`010fbec`](https://github.com/apache/spark/commit/010fbec39a3e1348529d2ecb0b5dbda51f0adee7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602522746
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24905/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602522081
**[Test build #120192 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120192/testReport)** for PR 27665 at commit [`010fbec`](https://github.com/apache/spark/commit/010fbec39a3e1348529d2ecb0b5dbda51f0adee7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602529630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24909/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396408948
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/TransportRequestHandler.java
##########
@@ -67,6 +68,7 @@ public TransportRequestHandler(
TransportClient reverseClient,
RpcHandler rpcHandler,
Long maxChunksBeingTransferred) {
+ super(reverseClient, rpcHandler.getStreamManager(), maxChunksBeingTransferred);
Review comment:
do we still need the `maxChunksBeingTransferred` variable in this class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676968
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120310/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
Victsm commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-590523197
@xuanyuanking we worked on the original fix of this issue. Having await in there is the key to the benefits provided in SPARK-24355, which improves reliability of Spark shuffle in a reasonable scaled deployment. This issue seems common across companies like us (LinkedIn), Netflix, Uber, Yahoo. As mentioned in #22173, what we observed is that in cases where HDD is used for shuffle storage, the disk is saturated first before the network can be saturated. So, for a reasonable scaled deployment, having this fix provides a boost in shuffle reliability without hurting much on the performance side. This is also validated by @tgravescs in the Yahoo deployment of this patch.
It's reasonable to introduce another config that disables this reliability improvement if it leads to performance regression in certain deployment mode. Just want to see whether we should leave this enabled by default or not. Also, as mentioned in #22173, we have discovered a potential fix to this perf regression issue that does not removes its reliability benefits. It will take some extra time on our side to evaluate that fix, which is a fix inside Netty. Want to make sure the broader community knows what we have been doing for this issue, so we do not take away a potential reliability improvement to Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603190468
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24984/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-604229321
Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603190468
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24984/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603190452
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602612162
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603654880
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602963063
**[Test build #120229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120229/testReport)** for PR 27665 at commit [`da8abd3`](https://github.com/apache/spark/commit/da8abd3dad70c4430053206ea07a459439840475).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589561436
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23524/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603192971
**[Test build #120274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120274/testReport)** for PR 27665 at commit [`34b52ef`](https://github.com/apache/spark/commit/34b52ef36420ecdbc9f70ee3dc36a3e72b29de1f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603003685
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-590495024
So I actually filed SPARK-30623 for this, please update the title and things. I don't think we need a separate feature config for this, as that jira mentioned I think was just say if the config isn't explicitly set then do synchronous mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676773
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120311/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676773
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120311/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603192971
**[Test build #120274 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120274/testReport)** for PR 27665 at commit [`34b52ef`](https://github.com/apache/spark/commit/34b52ef36420ecdbc9f70ee3dc36a3e72b29de1f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602612183
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120190/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-604240855
BTW, @xuanyuanking .
Could you confirm the above question?
- https://github.com/apache/spark/pull/27665#discussion_r398328411
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589638157
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/118772/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r397612491
##########
File path: common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java
##########
@@ -39,7 +39,7 @@
public class TransportRequestHandlerSuite {
@Test
- public void handleStreamRequest() {
+ public void handleStreamRequest() throws Exception {
Review comment:
Yes, because of the changes for `TransportRequestHandler.handler` here: https://github.com/apache/spark/pull/27665/files#diff-0e3429029d3f8d49e94ef11e4e3051a2R105
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396406401
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
what's wrong with the previous code?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602522746
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24905/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602963063
**[Test build #120229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120229/testReport)** for PR 27665 at commit [`da8abd3`](https://github.com/apache/spark/commit/da8abd3dad70c4430053206ea07a459439840475).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602518659
**[Test build #120190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120190/testReport)** for PR 27665 at commit [`5015f60`](https://github.com/apache/spark/commit/5015f60e0838a7703d6bf59700fc0ab38bc927cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602610010
**[Test build #120190 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120190/testReport)** for PR 27665 at commit [`5015f60`](https://github.com/apache/spark/commit/5015f60e0838a7703d6bf59700fc0ab38bc927cb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602630739
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120196/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r399010040
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
Thank you, @xuanyuanking and @cloud-fan .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602522738
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603003694
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120229/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602615330
**[Test build #120192 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120192/testReport)** for PR 27665 at commit [`010fbec`](https://github.com/apache/spark/commit/010fbec39a3e1348529d2ecb0b5dbda51f0adee7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602961327
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602529630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24909/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396858603
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -17,46 +17,33 @@
package org.apache.spark.network.server;
-import java.net.SocketAddress;
-
import com.google.common.base.Throwables;
-import io.netty.channel.Channel;
-import io.netty.channel.ChannelFuture;
-import io.netty.channel.ChannelFutureListener;
-import io.netty.channel.ChannelHandlerContext;
-import io.netty.channel.SimpleChannelInboundHandler;
+import io.netty.channel.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.spark.network.buffer.ManagedBuffer;
import org.apache.spark.network.client.TransportClient;
-import org.apache.spark.network.protocol.ChunkFetchFailure;
-import org.apache.spark.network.protocol.ChunkFetchRequest;
-import org.apache.spark.network.protocol.ChunkFetchSuccess;
-import org.apache.spark.network.protocol.Encodable;
+import org.apache.spark.network.protocol.*;
-import static org.apache.spark.network.util.NettyUtils.*;
+import static org.apache.spark.network.util.NettyUtils.getRemoteAddress;
/**
- * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
- * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
- * channel could potentially be blocked due to disk contentions. If several hundreds of clients
- * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
- * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
- * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
- * result, it would leave no threads left to process other RPC messages, which takes much less
- * time to process, and could lead to client timing out on either performing SASL authentication,
- * registering executors, or waiting for response for an OpenBlocks messages.
+ * Base class for custom ChannelHandler for FetchRequest messages.
+ * The messages can be processed in a separate event loop group or the one shared with other
+ * {@link Message.Type}s. The behavior is decided by whether the config
+ * `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
+ * See more details in SPARK-30623.
*/
-public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+public abstract class ChunkFetchRequestHandler
+ extends SimpleChannelInboundHandler<ChunkFetchRequest> {
private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
private final TransportClient client;
private final StreamManager streamManager;
- /** The max number of chunks being transferred and not finished yet. */
Review comment:
Sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602612162
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603649303
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603202850
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120273/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-590162209
cc @cloud-fan @tgravescs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r397102636
##########
File path: common/network-common/src/test/java/org/apache/spark/network/TransportRequestHandlerSuite.java
##########
@@ -39,7 +39,7 @@
public class TransportRequestHandlerSuite {
@Test
- public void handleStreamRequest() {
+ public void handleStreamRequest() throws Exception {
Review comment:
is this change necessary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-604253067
Thanks for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676958
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603650851
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603202843
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603697197
**[Test build #120326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120326/testReport)** for PR 27665 at commit [`4f42083`](https://github.com/apache/spark/commit/4f42083b43058d8f7276e5f095dc4b432c8b2b71).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603190452
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r398339001
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
because we only call this method if `separateChunkFetchRequest` returns true.
We will see exception if the assumption is broken.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603650519
**[Test build #120310 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120310/testReport)** for PR 27665 at commit [`34b52ef`](https://github.com/apache/spark/commit/34b52ef36420ecdbc9f70ee3dc36a3e72b29de1f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396410407
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -107,31 +85,10 @@ protected void channelRead0(
(ChannelFutureListener) future -> streamManager.chunkSent(msg.streamChunkId.streamId));
}
- /**
- * The invocation to channel.writeAndFlush is async, and the actual I/O on the
- * channel will be handled by the EventLoop the channel is registered to. So even
- * though we are processing the ChunkFetchRequest in a separate thread pool, the actual I/O,
- * which is the potentially blocking call that could deplete server handler threads, is still
- * being processed by TransportServer's default EventLoopGroup. In order to throttle the max
- * number of threads that channel I/O for sending response to ChunkFetchRequest, the thread
- * calling channel.writeAndFlush will wait for the completion of sending response back to
- * client by invoking await(). This will throttle the rate at which threads from
- * ChunkFetchRequest dedicated EventLoopGroup submit channel I/O requests to TransportServer's
- * default EventLoopGroup, thus making sure that we can reserve some threads in
- * TransportServer's default EventLoopGroup for handling other RPC messages.
- */
- private ChannelFuture respond(
- final Channel channel,
- final Encodable result) throws InterruptedException {
- final SocketAddress remoteAddress = channel.remoteAddress();
- return channel.writeAndFlush(result).await().addListener((ChannelFutureListener) future -> {
- if (future.isSuccess()) {
- logger.trace("Sent result {} to client {}", result, remoteAddress);
- } else {
- logger.error(String.format("Error sending result %s to %s; closing connection",
- result, remoteAddress), future.cause());
- channel.close();
- }
- });
- }
+ public abstract ChannelFuture respond(Channel channel, Encodable result) throws Exception;
Review comment:
why we need to define this method?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603694648
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603693966
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#discussion_r382763278
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -124,7 +127,13 @@ private ChannelFuture respond(
final Channel channel,
final Encodable result) throws InterruptedException {
final SocketAddress remoteAddress = channel.remoteAddress();
- return channel.writeAndFlush(result).await().addListener((ChannelFutureListener) future -> {
+ ChannelFuture channelFuture = null;
+ if (chunkFetchAsyncModeEnabled) {
+ channelFuture = channel.writeAndFlush(result);
Review comment:
When we remove `await`, we see more SASL requests timing out. I mentioned this in the comment here:
https://github.com/apache/spark/pull/22173#issuecomment-578347705
Wouldn't making the `asyncMode` default, make this problem worse?
In aysnc mode, how do you plan to tackle increased number of SASL failures?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r398340491
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
Review comment:
In this PR, we make the separate event loop group configurable by checking the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
> What do you mean by the config must be set
Here the function `chunkFetchHandlerThreads` is only called while the config is set. https://github.com/apache/spark/blob/0fe203e7032a326ff0c78f6660ab1b09409fe09d/common/network-common/src/main/java/org/apache/spark/network/TransportContext.java#L124-L129
> What value do you expect by default? Apparently, this seems to revert SPARK-25641 together without mentioning SPARK-25641.
Yes, this PR makes the feature disabled by default, let me also mention SPARK-25641 in PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589561429
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602525804
Do some refactoring to reuse the logic of processing fetch requests. Reuse the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` to control the separate event loop creation. Please take a look.
cc @otterc @Victsm @tgravescs @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602528973
**[Test build #120196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120196/testReport)** for PR 27665 at commit [`e655227`](https://github.com/apache/spark/commit/e65522707f9e8d96f65e279628661d138ffe8f55).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603003694
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120229/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603200510
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120274/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603694648
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602961331
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24942/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603650851
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602616600
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120192/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396855545
##########
File path: common/network-common/src/main/java/org/apache/spark/network/util/TransportConf.java
##########
@@ -339,12 +341,25 @@ public int chunkFetchHandlerThreads() {
return 0;
}
int chunkFetchHandlerThreadsPercent =
- conf.getInt("spark.shuffle.server.chunkFetchHandlerThreadsPercent", 100);
+ Integer.parseInt(conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent"));
int threads =
this.serverThreads() > 0 ? this.serverThreads() : 2 * NettyRuntime.availableProcessors();
return (int) Math.ceil(threads * (chunkFetchHandlerThreadsPercent / 100.0));
}
+ /**
+ * Whether to use a separate EventLoopGroup to process ChunkFetchRequest messages, it is decided
+ * by the config `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
+ */
+ public boolean separateChunkFetchRequest() {
+ try {
+ conf.get("spark.shuffle.server.chunkFetchHandlerThreadsPercent");
Review comment:
Yes, no API in `ConfigProvider`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589560868
**[Test build #118772 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118772/testReport)** for PR 27665 at commit [`743e566`](https://github.com/apache/spark/commit/743e566f952648be43fca1cc6e6781030651c1ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676394
**[Test build #120311 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120311/testReport)** for PR 27665 at commit [`4f42083`](https://github.com/apache/spark/commit/4f42083b43058d8f7276e5f095dc4b432c8b2b71).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676767
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603654887
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25021/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602961331
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24942/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602630739
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120196/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603676407
**[Test build #120310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120310/testReport)** for PR 27665 at commit [`34b52ef`](https://github.com/apache/spark/commit/34b52ef36420ecdbc9f70ee3dc36a3e72b29de1f).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602616600
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120192/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603202600
**[Test build #120273 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120273/testReport)** for PR 27665 at commit [`d4e0352`](https://github.com/apache/spark/commit/d4e0352ba54f5e05aae93b1e19183988e2d9f7e6).
* This patch **fails to generate documentation**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396406845
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/TransportChannelHandler.java
##########
@@ -58,19 +58,22 @@
private final TransportRequestHandler requestHandler;
private final long requestTimeoutNs;
private final boolean closeIdleConnections;
+ private final boolean separateChunkFetchRequest;
Review comment:
maybe a more explicit name: `skipChunkFetchRequest`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-604228688
it is now. merge script runs slowly at my sides...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#discussion_r396407166
##########
File path: common/network-common/src/main/java/org/apache/spark/network/server/ChunkFetchRequestHandler.java
##########
@@ -17,46 +17,33 @@
package org.apache.spark.network.server;
-import java.net.SocketAddress;
-
import com.google.common.base.Throwables;
-import io.netty.channel.Channel;
-import io.netty.channel.ChannelFuture;
-import io.netty.channel.ChannelFutureListener;
-import io.netty.channel.ChannelHandlerContext;
-import io.netty.channel.SimpleChannelInboundHandler;
+import io.netty.channel.*;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.spark.network.buffer.ManagedBuffer;
import org.apache.spark.network.client.TransportClient;
-import org.apache.spark.network.protocol.ChunkFetchFailure;
-import org.apache.spark.network.protocol.ChunkFetchRequest;
-import org.apache.spark.network.protocol.ChunkFetchSuccess;
-import org.apache.spark.network.protocol.Encodable;
+import org.apache.spark.network.protocol.*;
-import static org.apache.spark.network.util.NettyUtils.*;
+import static org.apache.spark.network.util.NettyUtils.getRemoteAddress;
/**
- * A dedicated ChannelHandler for processing ChunkFetchRequest messages. When sending response
- * of ChunkFetchRequest messages to the clients, the thread performing the I/O on the underlying
- * channel could potentially be blocked due to disk contentions. If several hundreds of clients
- * send ChunkFetchRequest to the server at the same time, it could potentially occupying all
- * threads from TransportServer's default EventLoopGroup for waiting for disk reads before it
- * can send the block data back to the client as part of the ChunkFetchSuccess messages. As a
- * result, it would leave no threads left to process other RPC messages, which takes much less
- * time to process, and could lead to client timing out on either performing SASL authentication,
- * registering executors, or waiting for response for an OpenBlocks messages.
+ * Base class for custom ChannelHandler for FetchRequest messages.
+ * The messages can be processed in a separate event loop group or the one shared with other
+ * {@link Message.Type}s. The behavior is decided by whether the config
+ * `spark.shuffle.server.chunkFetchHandlerThreadsPercent` is set or not.
+ * See more details in SPARK-30623.
*/
-public class ChunkFetchRequestHandler extends SimpleChannelInboundHandler<ChunkFetchRequest> {
+public abstract class ChunkFetchRequestHandler
+ extends SimpleChannelInboundHandler<ChunkFetchRequest> {
private static final Logger logger = LoggerFactory.getLogger(ChunkFetchRequestHandler.class);
private final TransportClient client;
private final StreamManager streamManager;
- /** The max number of chunks being transferred and not finished yet. */
Review comment:
shall we keep the comment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603189973
**[Test build #120273 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120273/testReport)** for PR 27665 at commit [`d4e0352`](https://github.com/apache/spark/commit/d4e0352ba54f5e05aae93b1e19183988e2d9f7e6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-602519280
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24903/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602528973
**[Test build #120196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120196/testReport)** for PR 27665 at commit [`e655227`](https://github.com/apache/spark/commit/e65522707f9e8d96f65e279628661d138ffe8f55).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602612183
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120190/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-24355][Core][FOLLOWUP] Add flag for fetching chunks in async mode
URL: https://github.com/apache/spark/pull/27665#issuecomment-589560868
**[Test build #118772 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/118772/testReport)** for PR 27665 at commit [`743e566`](https://github.com/apache/spark/commit/743e566f952648be43fca1cc6e6781030651c1ce).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-603697197
**[Test build #120326 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120326/testReport)** for PR 27665 at commit [`4f42083`](https://github.com/apache/spark/commit/4f42083b43058d8f7276e5f095dc4b432c8b2b71).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27665:
[SPARK-30623][Core] Spark external shuffle allow disable of separate event
loop group
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-602518659
**[Test build #120190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120190/testReport)** for PR 27665 at commit [`5015f60`](https://github.com/apache/spark/commit/5015f60e0838a7703d6bf59700fc0ab38bc927cb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27665: [SPARK-30623][Core]
Spark external shuffle allow disable of separate event loop group
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27665: [SPARK-30623][Core] Spark external shuffle allow disable of separate event loop group
URL: https://github.com/apache/spark/pull/27665#issuecomment-604215302
thanks, merging to master/3.0!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org