You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/01 23:37:38 UTC
[GitHub] [spark] hiboyang opened a new pull request #32031: [WIP ]Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
hiboyang opened a new pull request #32031:
URL: https://github.com/apache/spark/pull/32031
### What changes were proposed in this pull request?
This PR contains Remote Shuffle Service to support dynamic allocation on Kubernetes. The code is mostly copied from [Uber Remote Shuffle Service](https://github.com/uber/RemoteShuffleService) and modified with some renaming. Also added Kubernetes related support which does not exist in original Uber Remote Shuffle Service.
### Why are the changes needed?
It is still difficult to use dynamic allocation with Spark on Kubernetes. There are several disaggregated/remote shuffle solutions in different companies. Hopefully we could get a remote shuffle implementation into Spark and enhanced in the future by the Spark community.
### Does this PR introduce _any_ user-facing change?
Yes, user could set Spark config (spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager) to run Spark applications with remote shuffle service. It will make Spark use the new RssShuffleManager to write/read shuffle data to/from remote shuffle service.
### How was this patch tested?
Manually tested with Spark application in Kubernetes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816371258
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137106/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823027862
**[Test build #137675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137675/testReport)** for PR 32031 at commit [`ba7d5b6`](https://github.com/apache/spark/commit/ba7d5b604569012772c13511c6970e17985d1cdb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #32031:
URL: https://github.com/apache/spark/pull/32031#discussion_r681252225
##########
File path: remote-shuffle-service/src/main/java/org/apache/spark/remoteshuffle/StreamServerConfig.java
##########
@@ -0,0 +1,366 @@
+/*
+ * This file is copied from Uber Remote Shuffle Service
+(https://github.com/uber/RemoteShuffleService) and modified.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.remoteshuffle;
+
+import org.apache.spark.remoteshuffle.clients.ClientConstants;
+import org.apache.spark.remoteshuffle.execution.ShuffleExecutor;
+import org.apache.spark.remoteshuffle.handlers.UploadChannelManager;
+import org.apache.spark.remoteshuffle.metadata.ServiceRegistry;
+import org.apache.spark.remoteshuffle.storage.ShuffleFileStorage;
+import org.apache.spark.remoteshuffle.storage.ShuffleStorage;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.concurrent.TimeUnit;
+
+public class StreamServerConfig {
+ public static final long DEFAULT_SERVER_SIDE_CONNECTION_IDLE_TIMEOUT_MILLIS = 2 * 60 * 60 * 1000;
+
+ public static final String DEFAULT_DATA_CENTER = "dataCenter1";
+
+ private boolean useEpoll = false;
+
+ private int shufflePort = 19190;
+
+ private int httpPort = -1;
+
+ private String rootDir = "";
+
+ // number of threads for netty to accept incoming socket
+ private int nettyAcceptThreads = 2;
Review comment:
Curious to know why you have chosen number of threads in the boss group to be 2? Do you also run with this default in your production cluster (please ignore if this information cannot be shared)? I am asking because in the ESS the boss event group has just 1 thread and this isn't configurable. Is that something we should change for the ESS? I looked at the boss event-loop group a while back and found the threads in this group are responsible for accepting connection and assigning a worker thread to handle it (read and write to the socket connection). In general worker threads work for more time than the boss thread so it is not necessary to have more threads in the boss group. Did you face any issues with number of threads in this boss group being 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914536163
**[Test build #143058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143058/testReport)** for PR 32031 at commit [`3b2d36a`](https://github.com/apache/spark/commit/3b2d36a45b2c3682294c7acefdd5bec866f33b23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823946847
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137719/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866348616
**[Test build #140165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140165/testReport)** for PR 32031 at commit [`ff94688`](https://github.com/apache/spark/commit/ff9468849ecb4c615498b7b07fd6d5b09c61ba27).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825372353
**[Test build #137843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137843/testReport)** for PR 32031 at commit [`e5aae92`](https://github.com/apache/spark/commit/e5aae9263cf360025389e19b197e3753e496fd17).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825390526
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42373/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836229893
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42840/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826070220
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137877/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835070141
**[Test build #138276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138276/testReport)** for PR 32031 at commit [`7c20ac1`](https://github.com/apache/spark/commit/7c20ac129138065d0037df1d58fd16614f12f46f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #32031:
URL: https://github.com/apache/spark/pull/32031#discussion_r681252225
##########
File path: remote-shuffle-service/src/main/java/org/apache/spark/remoteshuffle/StreamServerConfig.java
##########
@@ -0,0 +1,366 @@
+/*
+ * This file is copied from Uber Remote Shuffle Service
+(https://github.com/uber/RemoteShuffleService) and modified.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.remoteshuffle;
+
+import org.apache.spark.remoteshuffle.clients.ClientConstants;
+import org.apache.spark.remoteshuffle.execution.ShuffleExecutor;
+import org.apache.spark.remoteshuffle.handlers.UploadChannelManager;
+import org.apache.spark.remoteshuffle.metadata.ServiceRegistry;
+import org.apache.spark.remoteshuffle.storage.ShuffleFileStorage;
+import org.apache.spark.remoteshuffle.storage.ShuffleStorage;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.conf.Configuration;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.concurrent.TimeUnit;
+
+public class StreamServerConfig {
+ public static final long DEFAULT_SERVER_SIDE_CONNECTION_IDLE_TIMEOUT_MILLIS = 2 * 60 * 60 * 1000;
+
+ public static final String DEFAULT_DATA_CENTER = "dataCenter1";
+
+ private boolean useEpoll = false;
+
+ private int shufflePort = 19190;
+
+ private int httpPort = -1;
+
+ private String rootDir = "";
+
+ // number of threads for netty to accept incoming socket
+ private int nettyAcceptThreads = 2;
Review comment:
Curious to know why you have chosen number of threads in the boss group to be 2? Do you also run with this default in your production cluster (please ignore if this information cannot be shared)? I am asking because in the ESS the boss event group has just 1 thread and this isn't configurable. Is that something we should change for the ESS? I looked at the boss event-loop group a while back and found the threads in this group are responsible for accepting connection and assigning a worker thread to handle it (read and write to the socket connection). In general worker threads work for more time than the boss thread. So, did you face any issues with number of threads in this boss group being 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862143106
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44380/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914536163
**[Test build #143058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143058/testReport)** for PR 32031 at commit [`3b2d36a`](https://github.com/apache/spark/commit/3b2d36a45b2c3682294c7acefdd5bec866f33b23).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914640286
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143058/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985941636
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812275259
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812665002
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914559984
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47561/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813824876
**[Test build #136932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136932/testReport)** for PR 32031 at commit [`e156dff`](https://github.com/apache/spark/commit/e156dff43345e9ca59f99b5cdf12f10963796a42).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836344087
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138318/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812266720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866348616
**[Test build #140165 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140165/testReport)** for PR 32031 at commit [`ff94688`](https://github.com/apache/spark/commit/ff9468849ecb4c615498b7b07fd6d5b09c61ba27).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835100566
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42798/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823864930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42247/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813848487
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835159459
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826043155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812685000
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41444/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hiboyang commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
hiboyang commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-815061331
Hi guys, thanks for the comments here! One of the purposes of this PR is to get more discussion and collect feedbacks/suggestions :) Hope it could serve as an example to show how remote shuffle service may work with Spark in Kubernetes. Putting it inside Spark repo will give people some hint about how shuffle manager interfaces may impact other shuffle solutions, and how to keep the shuffle interfaces flexible for other people to extend and implement their own shuffle algorithm.
This PR is still Work In Progress and we would like to see if any other people are interested to work together. We could have a zoom meeting to discuss more details as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866375260
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44691/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813838612
Also, +CC @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985928824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50394/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812271785
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935638380
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935672656
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812256562
**[Test build #136831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136831/testReport)** for PR 32031 at commit [`9f0bb75`](https://github.com/apache/spark/commit/9f0bb7519f044c7beee757ecaea73372cc252a0d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862077784
**[Test build #139851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139851/testReport)** for PR 32031 at commit [`c0de2d9`](https://github.com/apache/spark/commit/c0de2d97b7864accd4e0b3a3faf5b0ea255f03f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812730721
**[Test build #136871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136871/testReport)** for PR 32031 at commit [`44bcf80`](https://github.com/apache/spark/commit/44bcf802e7112c1bdd4dd29ca393da7a81938515).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823864930
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42247/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825444945
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835159459
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138276/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812768018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136871/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823831217
**[Test build #137719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137719/testReport)** for PR 32031 at commit [`a6a82d5`](https://github.com/apache/spark/commit/a6a82d5db28d5222b4ccc7b498e4f9d4c756a749).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862186552
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44380/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839624741
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138432/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826055997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823109088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826048576
**[Test build #137877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137877/testReport)** for PR 32031 at commit [`10945f2`](https://github.com/apache/spark/commit/10945f2d845b9f044acd65d40a278ced51acc2b9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826070220
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137877/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812275412
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835100566
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42798/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835149513
**[Test build #138276 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138276/testReport)** for PR 32031 at commit [`7c20ac1`](https://github.com/apache/spark/commit/7c20ac129138065d0037df1d58fd16614f12f46f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839620877
**[Test build #138432 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138432/testReport)** for PR 32031 at commit [`3729d48`](https://github.com/apache/spark/commit/3729d483d8cd4ff62ea42c3e56589f284c6b0854).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class CombinerRecordBufferManager[K, V, C](`
* `case class BufferManagerOptions(individualBufferSize: Int, individualBufferMax: Int,`
* `case class PartitionRecordBuffer(serializeStream: SerializationStream,`
* `class DefaultRecordBufferManager[K, V](serializer: Serializer,`
* `class KyroRecordBufferManager[K, V](serializerInstance: KryoSerializerInstance,`
* `trait RecordBufferManager[K, V] `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812236425
**[Test build #136827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136827/testReport)** for PR 32031 at commit [`af7a628`](https://github.com/apache/spark/commit/af7a628f44dee082aef0610f3131c30569007d0c).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class RssBuildInfo `
* `public class StreamServer `
* `public class StreamServerConfig `
* `public class BusyStatusSocketClient extends ClientBase `
* `public abstract class ClientBase implements AutoCloseable `
* `public class ClientConstants `
* `public class ClientRetryOptions `
* `public class DataBlockSocketReadClient extends ClientBase `
* `public class DataBlockSyncWriteClient extends ClientBase `
* `public class HeartbeatSocketClient extends ClientBase `
* `public class MultiServerAsyncWriteClient implements MultiServerWriteClient `
* `public class MultiServerHeartbeatClient implements AutoCloseable `
* `public class MultiServerSocketReadClient implements MultiServerReadClient `
* `public class MultiServerSyncWriteClient implements MultiServerWriteClient `
* `public class NotifyClient extends ClientBase `
* `public class PlainShuffleDataSocketReadClient extends ShuffleDataSocketReadClient `
* `public class PlainShuffleDataSyncWriteClient extends ShuffleDataSyncWriteClientBase `
* `public class PooledShuffleDataSyncWriteClient implements ShuffleDataSyncWriteClient `
* `public class PooledWriteClientFactory implements WriteClientFactory `
* `public class ReadClientDataOptions `
* `public class RegistryClient extends ClientBase `
* `public class ReplicatedReadClient implements MultiServerReadClient `
* `public class ReplicatedWriteClient implements MultiServerWriteClient `
* `public class RetriableSocketReadClient implements SingleServerReadClient `
* `public class ServerConnectionStringCache `
* `public class ServerIdAwareSocketReadClient implements SingleServerReadClient `
* `public class ServerIdAwareSyncWriteClient implements SingleServerWriteClient `
* `public class ServerReplicationGroupUtil `
* `public class ShuffleDataSocketReadClient implements AutoCloseable, SingleServerReadClient `
* `public abstract class ShuffleDataSyncWriteClientBase implements ShuffleDataSyncWriteClient `
* `public class ShuffleWriteConfig `
* `public class TaskDataBlock `
* `public class UnpooledWriteClientFactory implements WriteClientFactory `
* `public class AppMapId `
* `public class AppShuffleId `
* `public class AppShufflePartitionId `
* `public class AppTaskAttemptId `
* `public class Compression `
* `public class DataBlock `
* `public class DataBlockHeader `
* `public class DownloadServerVerboseInfo `
* `public class FilePathAndLength `
* `public class FixedLengthInputStream extends InputStream `
* `public class MapTaskCommitStatus `
* `public class MapTaskRssInfo `
* `public class MemoryMonitor `
* `public class PartitionFilePathAndLength `
* `public class ServerCandidate `
* `public class ServerDetail `
* `public class ServerDetailCollection `
* `public class ServerList `
* `public class ServerReplicationGroup `
* `public class ShuffleMapTaskAttemptId `
* `public class StreamServerMessageDecoder extends ByteToMessageDecoder `
* `public class StreamServerVersionDecoder extends ByteToMessageDecoder `
* `public class ExceptionWrapper<T extends Throwable> `
* `public class RssAggregateException extends RuntimeException `
* `public class RssDiskSpaceException extends RssException `
* `public class RssDuplicateAppTaskAttemptException extends RssException `
* `public class RssEndOfStreamException extends RssException `
* `public class RssException extends RuntimeException `
* `public class RssFileCorruptedException extends RssException `
* `public class RssFinishUploadException extends RssException `
* `public class RssInconsistentReplicaException extends RssException `
* `public class RssInvalidDataException extends RssException `
* `public class RssInvalidMapStatusException extends RssException `
* `public class RssInvalidServerIdException extends RssException `
* `public class RssInvalidStateException extends RssException `
* `public class RssMaxConnectionsException extends Exception `
* `public class RssMissingShuffleWriteConfigException extends RssException `
* `public class RssNetworkException extends RssException `
* `public class RssNoActiveReadClientException extends RssException `
* `public class RssNoServerAvailableException extends RssException `
* `public class RssNonRecoverableException extends RssException `
* `public class RssOperationQueueFullException extends RssException `
* `public class RssQueueNotReadyException extends RssException `
* `public class RssRetryTimeoutException extends RssException `
* `public class RssRetryableSparkTaskException extends RssException `
* `public class RssServerBusyException extends RssException `
* `public class RssServerDownException extends RssException `
* `public class RssServerResolveException extends RssException `
* `public class RssShuffleCorruptedException extends RssException `
* `public class RssShuffleDataNotAvailableException extends RssException `
* `public class RssShuffleStageNotStartedException extends RssException `
* `public class RssStaleTaskAttemptException extends RssException `
* `public class RssStreamReadException extends RssException `
* `public class RssTooMuchDataException extends RssException `
* `public class RssUberEnvironmentException extends RssException `
* `public class RssUnsupportedCompressionException extends RssException `
* `public class RssWriteRecordException extends RssException `
* `public class ExecutorAppState `
* `public class ExecutorShuffleStageState `
* `public class LocalFileStateStore implements StateStore `
* `public class LocalFileStateStoreIterator implements Iterator<BaseMessage>, AutoCloseable `
* `public class ShuffleDataWrapper `
* `public class ShuffleExecutor `
* `public class ShufflePartitionWriter `
* `public class StagePersistentInfo `
* `public class StateStoreLoadResult `
* `public class TaskAttemptCollection `
* `public class TaskAttemptIdAndState `
* `public class ChannelFutureCloseListener implements ChannelFutureListener `
* `public class ChannelIdleCheck implements Runnable `
* `public class DownloadChannelInboundHandler extends ChannelInboundHandlerAdapter `
* `public class DownloadServerHandler `
* `public class HandlerUtil `
* `public class HttpChannelInboundHandler extends ChannelInboundHandlerAdapter `
* `public class NotifyChannelInboundHandler extends ChannelInboundHandlerAdapter `
* `public class NotifyServerHandler `
* `public class RegistryChannelInboundHandler extends ChannelInboundHandlerAdapter `
* `public class RegistryServerHandler `
* `public class ResponseStatusAndMessage<T> `
* `public class UploadChannelInboundHandler extends ChannelInboundHandlerAdapter `
* `public class UploadChannelManager `
* `public class UploadServerHandler `
* `public class AppDeletionStateItem extends BaseMessage `
* `public abstract class BaseMessage extends SerializableMessage `
* `public class ConnectDownloadRequest extends BaseMessage `
* `public class ConnectDownloadResponse extends ServerResponseMessage `
* `public class ConnectNotifyRequest extends BaseMessage `
* `public class ConnectNotifyResponse extends ServerResponseMessage `
* `public class ConnectRegistryRequest extends BaseMessage `
* `public class ConnectRegistryResponse extends ServerResponseMessage `
* `public class ConnectUploadRequest extends BaseMessage `
* `public class ConnectUploadResponse extends ServerResponseMessage `
* `public abstract class ControlMessage extends BaseMessage `
* `public class FinishApplicationAttemptRequestMessage extends ControlMessage `
* `public class FinishApplicationJobRequestMessage extends ControlMessage `
* `public class FinishUploadMessage extends BaseMessage `
* `public class GetBusyStatusRequest extends BaseMessage `
* `public class GetBusyStatusResponse extends BaseMessage `
* `public class GetDataAvailabilityRequest extends BaseMessage `
* `public class GetDataAvailabilityResponse extends BaseMessage `
* `public class GetServersRequestMessage extends ControlMessage `
* `public class GetServersResponseMessage extends ServerResponseMessage `
* `public class HeartbeatMessage extends BaseMessage `
* `public class MessageConstants `
* `public class RegisterServerRequestMessage extends ControlMessage `
* `public class RegisterServerResponseMessage extends ServerResponseMessage `
* `public abstract class SerializableMessage `
* `public abstract class ServerResponseMessage extends BaseMessage `
* `public class ShuffleDataWrapper `
* `public class ShuffleStageStatus extends SerializableMessage `
* `public class StageCorruptionStateItem extends BaseMessage `
* `public class StageInfoStateItem extends BaseMessage `
* `public class StartUploadMessage extends BaseMessage `
* `public class TaskAttemptCommitStateItem extends BaseMessage `
* `public class InMemoryServiceRegistry implements ServiceRegistry `
* `public class ServerSequenceServiceRegistry implements ServiceRegistry `
* `public class ServiceRegistryUtils `
* `public class ServiceRegistryWrapper implements ServiceRegistry `
* `public class StandaloneServiceRegistryClient implements ServiceRegistry `
* `public class ApplicationJobStatusMetrics extends MetricGroup<ApplicationJobStatusMetricsKey> `
* `public class ApplicationJobStatusMetricsKey `
* `public class ApplicationMetrics extends MetricGroup<ApplicationMetricsKey> `
* `public class ApplicationMetricsKey `
* `public class ClientConnectMetrics extends MetricGroup<ClientConnectMetricsKey> `
* `public class ClientConnectMetricsKey `
* `public class ExceptionMetricGroupContainer `
* `public class ExceptionMetrics extends MetricGroup<ExceptionMetricsKey> `
* `public class ExceptionMetricsKey `
* `public class M3DummyScope implements Scope `
* `public class M3DummyScopeBuilder extends RootScopeBuilder `
* `public class M3Stats `
* ` .format(\"Failed to create ScopeBuilder instance from class name %s\",`
* `public class MetadataClientMetrics extends MetricGroup<MetadataClientMetricsKey> `
* `public class MetadataClientMetricsContainer `
* `public class MetadataClientMetricsKey `
* `public abstract class MetricGroup<K> implements AutoCloseable `
* `public class MetricGroupContainer<K, M extends MetricGroup<K>> `
* `public class NettyServerSideMetricGroupContainer<M extends MetricGroup<NettyServerSideMetricsKey>> `
* `public class NettyServerSideMetricsKey `
* `public class NotifyClientMetrics extends MetricGroup<NotifyClientMetricsKey> `
* `public class NotifyClientMetricsKey `
* `public class NotifyServerMetricsContainer `
* `public class ReadClientMetrics extends MetricGroup<ReadClientMetricsKey> `
* `public class ReadClientMetricsKey `
* `public class ScheduledMetricCollector `
* `public class ServerHandlerMetrics extends MetricGroup<NettyServerSideMetricsKey> `
* `public class ShuffleClientStageMetrics extends MetricGroup<ShuffleClientStageMetricsKey> `
* `public class ShuffleClientStageMetricsKey `
* `public class WriteClientMetrics extends MetricGroup<WriteClientMetricsKey> `
* `public class WriteClientMetricsKey `
* `public class ShuffleFileOutputStream implements ShuffleOutputStream `
* `public class ShuffleFileStorage implements ShuffleStorage `
* `public class ShuffleFileUtils `
* `public class FileDescriptorStressTest `
* `public class FsyncPerfTest `
* `public class PartitionFileChecker `
* `public class SerializerBenchmark `
* `public class StreamReadClientVerify `
* `public class StreamServerStressTool `
* `public class StreamServerStressToolLongRun `
* `public class StreamServerStressToolWrite64GB `
* `public class TestUtils `
* `public class AsyncSocketCompletionHandler implements CompletionHandler<Integer, AsyncSocketState> `
* `public class AsyncSocketState `
* `public class ByteBufUtils `
* `public class CountedOutputStream extends OutputStream `
* `public class ExceptionUtils `
* `public class FileUtils `
* `public class HttpUtils `
* `public class JsonUtils `
* `public class LogUtils `
* `public class MonitorUtils `
* `public class MovingAverageCalculator `
* `public class NettyUtils `
* `public class NetworkUtils `
* `public class ObjectWrapper<T> `
* `public class RateCounter `
* `public class RetryUtils `
* `public class ServerHostAndPort `
* `public class SocketUtils `
* `public class StreamUtils `
* `public class StringUtils `
* `public class SystemUtils `
* `public class ThreadUtils `
* `class MockTaskContext(val mockStageId: Int, val mockPartitionId: Int,`
* `class RssEmptyShuffleReader[K, C](`
* `case class RssServerSelectionResult(servers: Array[ServerDetail], replicas: Int,`
* `class RssShuffleBlockResolver extends ShuffleBlockResolver `
* `class RssShuffleManager(conf: SparkConf) extends ShuffleManager with Logging `
* `class RssShuffleReader[K, C](`
* `case class RssShuffleServerHandle(serverId: String, connectionString: String) `
* `class RssShuffleWriter[K, V, C](`
* `class BlockDownloaderPartitionRangeRecordIterator[K, C](`
* `class BlockDownloaderPartitionRecordIterator[K, C](`
* `class EmptyRecordIterator[K, C]() extends Iterator[Product2[K, C]] with Logging `
* `case class MapOutputRssInfo(numMaps: Int, numRssServers: Int, taskAttemptIds: Array[Long]) `
* `class RssSparkListener(val user: String, val appId: String, val attemptId: String,`
* `class RssStressTool extends Logging `
* `case class BufferManagerOptions(individualBufferSize: Int, individualBufferMax: Int,`
* `case class WriterBufferManagerValue(serializeStream: SerializationStream, output: Output)`
* `class WriteBufferManager(serializer: Serializer,`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837771488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835070141
**[Test build #138276 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138276/testReport)** for PR 32031 at commit [`7c20ac1`](https://github.com/apache/spark/commit/7c20ac129138065d0037df1d58fd16614f12f46f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812236256
**[Test build #136827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136827/testReport)** for PR 32031 at commit [`af7a628`](https://github.com/apache/spark/commit/af7a628f44dee082aef0610f3131c30569007d0c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813898617
**[Test build #136932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136932/testReport)** for PR 32031 at commit [`e156dff`](https://github.com/apache/spark/commit/e156dff43345e9ca59f99b5cdf12f10963796a42).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914607877
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47561/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826053752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826048520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935516370
**[Test build #143872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143872/testReport)** for PR 32031 at commit [`c411ff4`](https://github.com/apache/spark/commit/c411ff44fbbfd25564aac19d24df4bda52347694).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813846672
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813824876
**[Test build #136932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136932/testReport)** for PR 32031 at commit [`e156dff`](https://github.com/apache/spark/commit/e156dff43345e9ca59f99b5cdf12f10963796a42).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812256776
**[Test build #136831 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136831/testReport)** for PR 32031 at commit [`9f0bb75`](https://github.com/apache/spark/commit/9f0bb7519f044c7beee757ecaea73372cc252a0d).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826065128
**[Test build #137877 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137877/testReport)** for PR 32031 at commit [`10945f2`](https://github.com/apache/spark/commit/10945f2d845b9f044acd65d40a278ced51acc2b9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class MiscellaneousProcessDetails(`
* `case class SparkListenerMiscellaneousProcessAdded(time: Long, processId: String,`
* ` case class MiscellaneousProcessAdded(`
* `class ImmutableBitSet(val numBits: Int, val bitsToSet: Int*) extends BitSet(numBits) `
* `public class VectorizedBLAS extends F2jBLAS `
* `class PandasOnSparkFrameMethods(object):`
* `class PandasOnSparkSeriesMethods(object):`
* `class IndexOpsMixin(object, metaclass=ABCMeta):`
* `class CategoricalAccessor(object):`
* ` however, expected types are [(<class 'float'>, <class 'int'>)].`
* `class OptionError(AttributeError, KeyError):`
* `class DatetimeMethods(object):`
* `class DataError(Exception):`
* `class SparkPandasIndexingError(Exception):`
* `class SparkPandasNotImplementedError(NotImplementedError):`
* `class PandasNotImplementedError(NotImplementedError):`
* ` new_class = type(\"NameType\", (NameTypeHolder,), `
* ` new_class = type(\"NameType\", (NameTypeHolder,), `
* `class DataFrame(Frame, Generic[T]):`
* ` [defaultdict(<class 'list'>, `
* `defaultdict(<class 'list'>, `
* `class CachedDataFrame(DataFrame):`
* `class Frame(object, metaclass=ABCMeta):`
* `class GroupBy(object, metaclass=ABCMeta):`
* `class DataFrameGroupBy(GroupBy):`
* `class SeriesGroupBy(GroupBy):`
* `class Index(IndexOpsMixin):`
* `class CategoricalIndex(Index):`
* `class DatetimeIndex(Index):`
* `class MultiIndex(Index):`
* ` a single :class:`Index` (or subclass thereof).`
* `class NumericIndex(Index):`
* `class IntegerIndex(NumericIndex):`
* `class Int64Index(IntegerIndex):`
* `class Float64Index(NumericIndex):`
* `class IndexerLike(object):`
* `class AtIndexer(IndexerLike):`
* `class iAtIndexer(IndexerLike):`
* `class LocIndexerLike(IndexerLike, metaclass=ABCMeta):`
* `class LocIndexer(LocIndexerLike):`
* `class iLocIndexer(LocIndexerLike):`
* `class InternalFrame(object):`
* `class _MissingPandasLikeDataFrame(object):`
* `class MissingPandasLikeDataFrameGroupBy(object):`
* `class MissingPandasLikeSeriesGroupBy(object):`
* `class MissingPandasLikeIndex(object):`
* `class MissingPandasLikeDatetimeIndex(MissingPandasLikeIndex):`
* `class MissingPandasLikeCategoricalIndex(MissingPandasLikeIndex):`
* `class MissingPandasLikeMultiIndex(object):`
* `class MissingPandasLikeSeries(object):`
* `class MissingPandasLikeExpanding(object):`
* `class MissingPandasLikeRolling(object):`
* `class MissingPandasLikeExpandingGroupby(object):`
* `class MissingPandasLikeRollingGroupby(object):`
* `class PythonModelWrapper(object):`
* `class PandasOnSparkPlotAccessor(PandasObject):`
* `class PandasOnSparkBarPlot(PandasBarPlot, TopNPlotBase):`
* `class PandasOnSparkBoxPlot(PandasBoxPlot, BoxPlotBase):`
* `class PandasOnSparkHistPlot(PandasHistPlot, HistogramPlotBase):`
* `class PandasOnSparkPiePlot(PandasPiePlot, TopNPlotBase):`
* `class PandasOnSparkAreaPlot(PandasAreaPlot, SampledPlotBase):`
* `class PandasOnSparkLinePlot(PandasLinePlot, SampledPlotBase):`
* `class PandasOnSparkBarhPlot(PandasBarhPlot, TopNPlotBase):`
* `class PandasOnSparkScatterPlot(PandasScatterPlot, TopNPlotBase):`
* `class PandasOnSparkKdePlot(PandasKdePlot, KdePlotBase):`
* ` new_class = type(\"NameType\", (NameTypeHolder,), `
* ` new_class = param.type if isinstance(param, np.dtype) else param`
* `class Series(Frame, IndexOpsMixin, Generic[T]):`
* ` dictionary is a ``dict`` subclass that defines ``__missing__`` (i.e.`
* ` defaultdict(<class 'list'>, `
* `class SparkIndexOpsMethods(object, metaclass=ABCMeta):`
* `class SparkSeriesMethods(SparkIndexOpsMethods):`
* `class SparkIndexMethods(SparkIndexOpsMethods):`
* `class SparkFrameMethods(object):`
* `class CachedSparkFrameMethods(SparkFrameMethods):`
* `class SQLProcessor(object):`
* `class StringMethods(object):`
* `class SeriesType(Generic[T]):`
* `class DataFrameType(object):`
* `class ScalarType(object):`
* `class UnknownType(object):`
* `class NameTypeHolder(object):`
* ` The returned type class indicates both dtypes (a pandas only dtype object`
* `class PandasOnSparkUsageLogger(object):`
* `class RollingAndExpanding(object):`
* `class Rolling(RollingAndExpanding):`
* `class RollingGroupby(Rolling):`
* `class Expanding(RollingAndExpanding):`
* `class ExpandingGroupby(Expanding):`
* `trait FunctionRegistryBase[T] `
* `trait SimpleFunctionRegistryBase[T] extends FunctionRegistryBase[T] with Logging `
* `trait EmptyFunctionRegistryBase[T] extends FunctionRegistryBase[T] `
* `trait FunctionRegistry extends FunctionRegistryBase[Expression] `
* `trait TableFunctionRegistry extends FunctionRegistryBase[LogicalPlan] `
* `class NoSuchFunctionException(`
* `case class ResolveTableValuedFunctions(catalog: SessionCatalog) extends Rule[LogicalPlan] `
* ` case class CombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends TypeCoercionRule `
* `abstract class QuaternaryExpression extends Expression with QuaternaryLike[Expression] `
* `abstract class Covariance(val left: Expression, val right: Expression, nullOnDivideByZero: Boolean)`
* `case class KnownFloatingPointNormalized(child: Expression) extends TaggingExpression `
* `trait BaseGroupingSets extends Expression with CodegenFallback `
* `case class Cube(`
* `trait SimpleHigherOrderFunction extends HigherOrderFunction with BinaryLike[Expression] `
* `case class Acos(child: Expression) extends UnaryMathExpression(math.acos, \"ACOS\") `
* `case class Asin(child: Expression) extends UnaryMathExpression(math.asin, \"ASIN\") `
* `case class Atan(child: Expression) extends UnaryMathExpression(math.atan, \"ATAN\") `
* `case class Cbrt(child: Expression) extends UnaryMathExpression(math.cbrt, \"CBRT\") `
* `case class Cos(child: Expression) extends UnaryMathExpression(math.cos, \"COS\") `
* `case class Cosh(child: Expression) extends UnaryMathExpression(math.cosh, \"COSH\") `
* `case class Log10(child: Expression) extends UnaryLogExpression(StrictMath.log10, \"LOG10\") `
* `case class Signum(child: Expression) extends UnaryMathExpression(math.signum, \"SIGNUM\") `
* `case class Sin(child: Expression) extends UnaryMathExpression(math.sin, \"SIN\") `
* `case class Sinh(child: Expression) extends UnaryMathExpression(math.sinh, \"SINH\") `
* `case class Sqrt(child: Expression) extends UnaryMathExpression(math.sqrt, \"SQRT\") `
* `case class Tan(child: Expression) extends UnaryMathExpression(math.tan, \"TAN\") `
* `case class Tanh(child: Expression) extends UnaryMathExpression(math.tanh, \"TANH\") `
* `trait AnalysisOnlyCommand extends Command `
* `case class DomainJoin(domainAttrs: Seq[Attribute], child: LogicalPlan) extends UnaryNode `
* `case class DeleteAction(condition: Option[Expression]) extends MergeAction `
* `case class UpdateStarAction(condition: Option[Expression]) extends MergeAction `
* `case class InsertStarAction(condition: Option[Expression]) extends MergeAction `
* `case class RefreshTable(child: LogicalPlan) extends UnaryCommand `
* `case class CommentOnNamespace(child: LogicalPlan, comment: String) extends UnaryCommand `
* `case class CommentOnTable(child: LogicalPlan, comment: String) extends UnaryCommand `
* `case class RefreshFunction(child: LogicalPlan) extends UnaryCommand `
* `case class DescribeFunction(child: LogicalPlan, isExtended: Boolean) extends UnaryCommand `
* `case class RecoverPartitions(child: LogicalPlan) extends UnaryCommand `
* `case class RuleId(id: Int) `
* `abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product with TreePatternBits `
* `trait QuaternaryLike[T <: TreeNode[T]] `
* `trait TreePatternBits `
* ` implicit class MetadataColumnHelper(attr: Attribute) `
* `public class OrcArrayColumnVector extends OrcColumnVector `
* `public class OrcAtomicColumnVector extends OrcColumnVector `
* `public abstract class OrcColumnVector extends org.apache.spark.sql.vectorized.ColumnVector `
* `class OrcColumnVectorUtils `
* `public class OrcMapColumnVector extends OrcColumnVector `
* `public class OrcStructColumnVector extends OrcColumnVector `
* `trait DataWritingCommand extends UnaryCommand `
* `case class SetCommand(kv: Option[(String, Option[String])])`
* `case class ResetCommand(config: Option[String]) extends LeafRunnableCommand with IgnoreCachedData `
* `trait RunnableCommand extends Command `
* `case class AddJarCommand(path: String) extends LeafRunnableCommand `
* `case class AddFileCommand(path: String) extends LeafRunnableCommand `
* `case class AddArchiveCommand(path: String) extends LeafRunnableCommand `
* `case class ListFilesCommand(files: Seq[String] = Seq.empty[String]) extends LeafRunnableCommand `
* `case class ListJarsCommand(jars: Seq[String] = Seq.empty[String]) extends LeafRunnableCommand `
* `case class ListArchivesCommand(archives: Seq[String] = Seq.empty[String])`
* `abstract class DescribeCommandBase extends LeafRunnableCommand `
* `trait BaseCacheTableExec extends LeafV2CommandExec `
* `sealed trait V1FallbackWriters extends LeafV2CommandExec with SupportsV1Write `
* `case class WriteToDataSourceV2(`
* `case class LocalLimitExec(limit: Int, child: SparkPlan) extends BaseLimitExec `
* `abstract class CustomSumMetric extends CustomMetric `
* `abstract class CustomAvgMetric extends CustomMetric `
* `case class WriteToMicroBatchDataSource(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985906774
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50394/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985920414
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50394/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985941636
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145918/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836304888
**[Test build #138318 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138318/testReport)** for PR 32031 at commit [`506149f`](https://github.com/apache/spark/commit/506149f3fa92b27bdf09da6748e91516b6dd5aea).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RssWritePerfTool extends ShuffleWritePerfTool with Logging `
* `class ShuffleWritePerfToolPartitioner(val partitions: Int) extends Partitioner `
* `abstract class ShuffleWritePerfTool extends Logging `
* `class SortMergeWritePerfTool extends ShuffleWritePerfTool with Logging `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935730546
**[Test build #143872 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143872/testReport)** for PR 32031 at commit [`c411ff4`](https://github.com/apache/spark/commit/c411ff44fbbfd25564aac19d24df4bda52347694).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826048520
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42405/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813838612
Also, +CC @Ngone51, @attilapiros, @otterc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823831217
**[Test build #137719 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137719/testReport)** for PR 32031 at commit [`a6a82d5`](https://github.com/apache/spark/commit/a6a82d5db28d5222b4ccc7b498e4f9d4c756a749).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862186552
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44380/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823027862
**[Test build #137675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137675/testReport)** for PR 32031 at commit [`ba7d5b6`](https://github.com/apache/spark/commit/ba7d5b604569012772c13511c6970e17985d1cdb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812743893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816340022
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823077716
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826055997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812680029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-814126917
yeah its unclear to me how this fits in with the rest of the shuffle work going on and to be honest I haven't kept up enough to comment without reviewing that again to see where everything is at. This is also just a huge PR to review in one shot.
I think it would be nice to have a remote shuffle service solution for k8s in spark but it should also be pluggable and we should chose what that looks like. My reasons are:
- we have one for other cluster managers and having one for k8s allows us to make sure it integrates well with everything like dynamic allocation and other features. The version internally should use the same plugin api others use.
- ease of use
- make sure its tested
- ability to more easily change it, enhance it and keep it in sync. If there are other ones great, but many times those are in private repos and hard to get changes in.
I agree with the point as to why this one should go in and I think we should discuss and see what makes the most sense. In some ways you also have to look at who is going to put in the effort to get it merged and also maintain it. Shuffle work has been going on for a long time now and it seems like we get a few prs and then stops for a while. Obviously people have other things going on too and its a lot of effort.
In my mind it comes down is it beneficial to the majority of users, who is going to put in the effort and is it going to be maintained if its pulled in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836185216
**[Test build #138318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138318/testReport)** for PR 32031 at commit [`506149f`](https://github.com/apache/spark/commit/506149f3fa92b27bdf09da6748e91516b6dd5aea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862077784
**[Test build #139851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139851/testReport)** for PR 32031 at commit [`c0de2d9`](https://github.com/apache/spark/commit/c0de2d97b7864accd4e0b3a3faf5b0ea255f03f1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985928824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/50394/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862167911
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44380/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914584748
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47561/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823120150
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137675/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826048520
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812256859
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837771488
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816316433
**[Test build #137106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137106/testReport)** for PR 32031 at commit [`c46fe43`](https://github.com/apache/spark/commit/c46fe43ea285ec49998bda6a7b6b8ad54a12acfd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826063002
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826048520
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42405/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836228137
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42840/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812236434
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136827/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839533740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42953/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823082430
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839533691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825372353
**[Test build #137843 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137843/testReport)** for PR 32031 at commit [`e5aae92`](https://github.com/apache/spark/commit/e5aae9263cf360025389e19b197e3753e496fd17).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812761051
**[Test build #136871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136871/testReport)** for PR 32031 at commit [`44bcf80`](https://github.com/apache/spark/commit/44bcf802e7112c1bdd4dd29ca393da7a81938515).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812664990
**[Test build #136866 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136866/testReport)** for PR 32031 at commit [`0222598`](https://github.com/apache/spark/commit/022259831e0c77e9aaa326eebe38b7caf1bf8c14).
* This patch **fails RAT tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812274710
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823119110
**[Test build #137675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137675/testReport)** for PR 32031 at commit [`ba7d5b6`](https://github.com/apache/spark/commit/ba7d5b604569012772c13511c6970e17985d1cdb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RecordCombinedSerializationBuffer[K, V, C](`
* `trait RecordSerializationBuffer[K, V] `
* `class WriteBufferManager[K, V](serializer: Serializer,`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813918553
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136932/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812979950
BTW, thank you for making a PR, @hiboyang . 😄
> Sounds good, thanks Dongjoon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837901710
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138353/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836224236
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42840/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hiboyang commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
hiboyang commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812963591
> I removed `to support dynamic allocation` from the PR title because Apache Spark has been supporting Dynamic allocation in K8s environment since 3.0.0.
>
> * http://spark.apache.org/releases/spark-release-3-0-0.html
Sounds good, thanks Dongjoon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935516370
**[Test build #143872 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143872/testReport)** for PR 32031 at commit [`c411ff4`](https://github.com/apache/spark/commit/c411ff44fbbfd25564aac19d24df4bda52347694).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826047487
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823120150
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137675/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812979974
cc @holdenk and @attilapiros
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826043155
**[Test build #137875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137875/testReport)** for PR 32031 at commit [`cf49308`](https://github.com/apache/spark/commit/cf493080b1b9db3e8eac8caa291b666c72325e3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812236434
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136827/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826058834
**[Test build #137875 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137875/testReport)** for PR 32031 at commit [`cf49308`](https://github.com/apache/spark/commit/cf493080b1b9db3e8eac8caa291b666c72325e3f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914635595
**[Test build #143058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143058/testReport)** for PR 32031 at commit [`3b2d36a`](https://github.com/apache/spark/commit/3b2d36a45b2c3682294c7acefdd5bec866f33b23).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914607877
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47561/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825390526
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42373/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-1072899711
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935672656
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837717297
**[Test build #138353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138353/testReport)** for PR 32031 at commit [`48a0be7`](https://github.com/apache/spark/commit/48a0be7b58a1a9d5c556b35359427fb0cd903b13).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812256859
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136831/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812685000
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41444/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812745539
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41449/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866379920
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44691/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837873050
**[Test build #138353 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138353/testReport)** for PR 32031 at commit [`48a0be7`](https://github.com/apache/spark/commit/48a0be7b58a1a9d5c556b35359427fb0cd903b13).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839533740
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42953/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812275412
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41412/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866352620
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140165/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836185216
**[Test build #138318 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138318/testReport)** for PR 32031 at commit [`506149f`](https://github.com/apache/spark/commit/506149f3fa92b27bdf09da6748e91516b6dd5aea).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812768018
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136871/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935577341
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48384/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813820509
I am assuming this is going to be built on top of SPARK-25299 ?
Given SPARK-25299 is still not in, and has major interfaces still in design ... would be a good idea to converge there before ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-868733992
Thanks for working on this, the PR is super huge though is it possible to break into smaller chunks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823945133
**[Test build #137719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137719/testReport)** for PR 32031 at commit [`a6a82d5`](https://github.com/apache/spark/commit/a6a82d5db28d5222b4ccc7b498e4f9d4c756a749).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RecordPlainSerializationBuffer[K, V](`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826043155
**[Test build #137875 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137875/testReport)** for PR 32031 at commit [`cf49308`](https://github.com/apache/spark/commit/cf493080b1b9db3e8eac8caa291b666c72325e3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816338181
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41684/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825385480
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866379893
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44691/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839624741
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138432/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816340022
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41684/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837755790
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816316433
**[Test build #137106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137106/testReport)** for PR 32031 at commit [`c46fe43`](https://github.com/apache/spark/commit/c46fe43ea285ec49998bda6a7b6b8ad54a12acfd).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825423428
**[Test build #137843 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137843/testReport)** for PR 32031 at commit [`e5aae92`](https://github.com/apache/spark/commit/e5aae9263cf360025389e19b197e3753e496fd17).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813848487
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41509/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837901710
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138353/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862197777
**[Test build #139851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139851/testReport)** for PR 32031 at commit [`c0de2d9`](https://github.com/apache/spark/commit/c0de2d97b7864accd4e0b3a3faf5b0ea255f03f1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823109088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42203/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816340011
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41684/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812256562
**[Test build #136831 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136831/testReport)** for PR 32031 at commit [`9f0bb75`](https://github.com/apache/spark/commit/9f0bb7519f044c7beee757ecaea73372cc252a0d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835098145
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42798/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826048018
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42405/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836229893
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42840/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812271785
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41407/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812664757
**[Test build #136866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136866/testReport)** for PR 32031 at commit [`0222598`](https://github.com/apache/spark/commit/022259831e0c77e9aaa326eebe38b7caf1bf8c14).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812665002
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136866/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812664757
**[Test build #136866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136866/testReport)** for PR 32031 at commit [`0222598`](https://github.com/apache/spark/commit/022259831e0c77e9aaa326eebe38b7caf1bf8c14).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839499587
**[Test build #138432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138432/testReport)** for PR 32031 at commit [`3729d48`](https://github.com/apache/spark/commit/3729d483d8cd4ff62ea42c3e56589f284c6b0854).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-836344087
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138318/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985888716
**[Test build #145918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145918/testReport)** for PR 32031 at commit [`2300dc4`](https://github.com/apache/spark/commit/2300dc405206e053e38ea2ccdb5936969c2271ef).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935764953
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-935764953
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143872/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-814047657
I think the followings are good reasons to keep `remote-shuffle-service` as a separate plugin:
- most of the changes are brand new files added under directory `remote-shuffle-service` (the exceptions are only two `pom.xml` files: on in the root and the other in the assembly directory)
- it is even deployed separately
- as you wrote `There are several disaggregated/remote shuffle solutions in different companies`, so why this one is chosen to be included? (although the rest of the solutions are not open sourced or open sourced yet)
- if it remains as separate plugin it can be released separately (not depending on Spark releases might be needed as this code is fresh and not as tested so probably much more releases will be needed from the `remote-shuffle-service` in the beginning than from Spark)
- as a separate thing the borders are more clean and easier to be kept clean
- as a separate plugin the developer who knows everything about it has full control over the source code
What would be advantage to have `remote-shuffle-service` be integrated into Spark?
I can only see one advantage: this solution would be more advertised as the one which officially included.
But I think widespread usage have to be achieved differently: by making remote-shuffle-service much more easier to be used on k8s is good direction. In addition to the docker image build provided here by giving scripts, tools and guidance how to set it up and configure it for your Spark jobs.
As more and more users will use it more and more experiences we will have and definitely more developers will join to work on it.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862199008
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139851/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812236256
**[Test build #136827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136827/testReport)** for PR 32031 at commit [`af7a628`](https://github.com/apache/spark/commit/af7a628f44dee082aef0610f3131c30569007d0c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-914640286
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143058/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826047487
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42405/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hiboyang commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
hiboyang commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-868866412
> Thanks for working on this, the PR is super huge though is it possible to break into smaller chunks?
Yes, it is good suggestion! Let me see how I could break this into smaller chunks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826048576
**[Test build #137877 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137877/testReport)** for PR 32031 at commit [`10945f2`](https://github.com/apache/spark/commit/10945f2d845b9f044acd65d40a278ced51acc2b9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812745539
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41449/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-826063002
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137875/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-816361839
**[Test build #137106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137106/testReport)** for PR 32031 at commit [`c46fe43`](https://github.com/apache/spark/commit/c46fe43ea285ec49998bda6a7b6b8ad54a12acfd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866352580
**[Test build #140165 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140165/testReport)** for PR 32031 at commit [`ff94688`](https://github.com/apache/spark/commit/ff9468849ecb4c615498b7b07fd6d5b09c61ba27).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RssShuffleManagerWithServerSequenceRegistryTest `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-825444945
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137843/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823864748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service to support dynamic allocation on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-812730721
**[Test build #136871 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136871/testReport)** for PR 32031 at commit [`44bcf80`](https://github.com/apache/spark/commit/44bcf802e7112c1bdd4dd29ca393da7a81938515).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-813918553
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136932/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866379920
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44691/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-862199008
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139851/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-835100545
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42798/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-866352620
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140165/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-823946847
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137719/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-839499587
**[Test build #138432 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138432/testReport)** for PR 32031 at commit [`3729d48`](https://github.com/apache/spark/commit/3729d483d8cd4ff62ea42c3e56589f284c6b0854).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985939585
**[Test build #145918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145918/testReport)** for PR 32031 at commit [`2300dc4`](https://github.com/apache/spark/commit/2300dc405206e053e38ea2ccdb5936969c2271ef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class NettyLogger `
* `public final class LZ4Compressor `
* `public final class LZ4Factory `
* `public final class LZ4SafeDecompressor `
* `public class JavaModuleOptions `
* `class SparkConf(object):`
* `class ProbabilisticClassifier(Classifier, _ProbabilisticClassifierParams, metaclass=ABCMeta):`
* `class ProbabilisticClassificationModel(`
* `class _JavaProbabilisticClassifier(ProbabilisticClassifier, _JavaClassifier, metaclass=ABCMeta):`
* `class _JavaProbabilisticClassificationModel(`
* `class _LinearSVCParams(`
* `class LinearSVCModel(`
* `class _LogisticRegressionParams(`
* `class LogisticRegression(`
* `class LogisticRegressionModel(`
* `class BinaryLogisticRegressionSummary(_BinaryClassificationSummary, LogisticRegressionSummary):`
* `class BinaryLogisticRegressionTrainingSummary(`
* `class DecisionTreeClassifier(`
* `class DecisionTreeClassificationModel(`
* `class RandomForestClassifier(`
* `class RandomForestClassificationModel(`
* `class RandomForestClassificationTrainingSummary(`
* `class BinaryRandomForestClassificationTrainingSummary(`
* `class GBTClassifier(`
* `class GBTClassificationModel(`
* `class NaiveBayes(`
* `class NaiveBayesModel(`
* `class _MultilayerPerceptronParams(`
* `class MultilayerPerceptronClassifier(`
* `class MultilayerPerceptronClassificationModel(`
* `class MultilayerPerceptronClassificationTrainingSummary(`
* `class FMClassifier(`
* `class FMClassificationModel(`
* `class _GaussianMixtureParams(`
* `class GaussianMixtureModel(`
* `class _KMeansParams(`
* `class KMeansModel(`
* `class _BisectingKMeansParams(`
* `class BisectingKMeansModel(`
* `class PowerIterationClustering(`
* `class BinaryClassificationEvaluator(`
* `class RegressionEvaluator(`
* `class MulticlassClassificationEvaluator(`
* `class MultilabelClassificationEvaluator(`
* `class ClusteringEvaluator(`
* `class RankingEvaluator(`
* `class Binarizer(`
* `class BucketedRandomProjectionLSH(`
* `class BucketedRandomProjectionLSHModel(`
* `class Bucketizer(`
* `class ElementwiseProduct(`
* `class FeatureHasher(`
* `class HashingTF(`
* `class _OneHotEncoderParams(`
* `class PolynomialExpansion(`
* `class QuantileDiscretizer(`
* `class _StringIndexerParams(`
* `class StopWordsRemover(`
* `class VectorAssembler(`
* `class VectorSizeHint(`
* `class VarianceThresholdSelector(`
* `class VarianceThresholdSelectorModel(`
* `class UnivariateFeatureSelector(`
* `class UnivariateFeatureSelectorModel(`
* `class _LinearRegressionParams(`
* `class LinearRegressionModel(`
* `class IsotonicRegression(`
* `class IsotonicRegressionModel(JavaModel, _IsotonicRegressionParams, JavaMLWritable, JavaMLReadable):`
* `class DecisionTreeRegressor(`
* `class RandomForestRegressor(`
* `class _AFTSurvivalRegressionParams(`
* `class AFTSurvivalRegression(`
* `class AFTSurvivalRegressionModel(`
* `class _GeneralizedLinearRegressionParams(`
* `class GeneralizedLinearRegression(`
* `class GeneralizedLinearRegressionModel(`
* `class _FactorizationMachinesParams(`
* `class FMRegressionModel(`
* `class CrossValidator(`
* `class TrainValidationSplit(`
* ` + \"class name `
* `class MultivariateGaussian(NamedTuple):`
* `class DatetimeNTZOps(DatetimeOps):`
* `class TimedeltaOps(DataTypeOps):`
* `class TimedeltaIndex(Index):`
* `class MissingPandasLikeTimedeltaIndex(MissingPandasLikeIndex):`
* `class MissingPandasLikeMultiIndex(object):`
* `class HistogramPlotBase(NumericPlotBase):`
* `class KdePlotBase(NumericPlotBase):`
* `class SQLStringFormatter(string.Formatter):`
* `class IndexNameTypeHolder(object):`
* ` new_class = type(NameTypeHolder.short_name, (NameTypeHolder,), `
* ` new_class = param.type if isinstance(param, np.dtype) else param`
* `class PandasAPIOnSparkAdviceWarning(Warning):`
* `class UDFBasicProfiler(BasicProfiler):`
* `class ResultIterable(Iterable[T]):`
* `class CloudPickleSerializer(FramedSerializer):`
* `class Database(NamedTuple):`
* `class Table(NamedTuple):`
* `class Column(NamedTuple):`
* `class Function(NamedTuple):`
* `class ArrowStreamUDFSerializer(ArrowStreamSerializer):`
* `class OptionUtils(object):`
* `class TimestampNTZType(AtomicType, metaclass=DataTypeSingleton):`
* `class DayTimeIntervalType(AtomicType):`
* `class DatetimeNTZConverter(object):`
* `class DayTimeIntervalTypeConverter(object):`
* `class SparkUpgradeException(CapturedException):`
* `class SparkJobInfo(NamedTuple):`
* `class SparkStageInfo(NamedTuple):`
* `class ExecutorPodsPollingSnapshotSource(`
* `class ExecutorPodsWatchSnapshotSource(`
* ` protected class YarnSchedulerEndpoint(override val rpcEnv: RpcEnv)`
* `public class ExpressionImplUtils `
* `public final class TableIndex `
* `public final class AlwaysFalse extends Filter `
* `public final class AlwaysTrue extends Filter `
* `public final class And extends BinaryFilter `
* `abstract class BinaryComparison extends Filter `
* `abstract class BinaryFilter extends Filter `
* `public final class EqualNullSafe extends BinaryComparison `
* `public final class EqualTo extends BinaryComparison `
* `public abstract class Filter implements Expression, Serializable `
* `public final class GreaterThan extends BinaryComparison `
* `public final class GreaterThanOrEqual extends BinaryComparison `
* `public final class In extends Filter `
* `public final class IsNotNull extends Filter `
* `public final class IsNull extends Filter `
* `public final class LessThan extends BinaryComparison `
* `public final class LessThanOrEqual extends BinaryComparison `
* `public final class Not extends Filter `
* `public final class Or extends BinaryFilter `
* `public final class StringContains extends StringPredicate `
* `public final class StringEndsWith extends StringPredicate `
* `abstract class StringPredicate extends Filter `
* `public final class StringStartsWith extends StringPredicate `
* `public class NumericHistogram `
* ` * The Coord class defines a histogram bin, which is just an (x,y) pair.`
* ` public static class Coord implements Comparable `
* `public class ColumnarBatch implements AutoCloseable `
* `public final class ColumnarBatchRow extends InternalRow `
* `class IndexAlreadyExistsException(message: String, cause: Option[Throwable] = None)`
* ` class AnsiCombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends`
* `trait ExpressionBuilder `
* `class NoSuchIndexException(message: String, cause: Option[Throwable] = None)`
* `case class RelationTimeTravel(`
* `case class AsOfTimestamp(timestamp: Long) extends TimeTravelSpec`
* `case class AsOfVersion(version: String) extends TimeTravelSpec`
* ` class CombinedTypeCoercionRule(rules: Seq[TypeCoercionRule]) extends TypeCoercionRule `
* `trait FunctionExpressionBuilder `
* `case class ExpressionStats(expr: Expression)(var useCount: Int)`
* `case class PrettyPythonUDF(`
* `case class HistogramNumeric(`
* `trait ExtractValue extends Expression with NullIntolerant `
* `case class CastTimestampNTZToLong(child: Expression) extends TimestampToLongBase `
* `case class Sec(child: Expression)`
* `case class Csc(child: Expression)`
* `case class AesEncrypt(input: Expression, key: Expression, child: Expression)`
* `case class AesDecrypt(input: Expression, key: Expression, child: Expression)`
* `case class ILike(`
* `trait PadExpressionBuilderBase extends ExpressionBuilder `
* `case class StringLPad(str: Expression, len: Expression, pad: Expression)`
* `case class BinaryLPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class BinaryRPad(str: Expression, len: Expression, pad: Expression, child: Expression)`
* `case class UnclosedCommentProcessor(`
* `trait OperationHelper extends AliasHelper with PredicateHelper `
* `case class AsOfJoin(`
* `case class PythonMapInArrow(`
* `case class CreateTable(`
* `case class SetCatalogAndNamespace(child: LogicalPlan) extends UnaryCommand `
* `case class CreateFunction(`
* `case class CreateView(`
* `case class CreateIndex(`
* `case class DropIndex(`
* `case class TableSpec(`
* `trait SupportsPushDownCatalystFilters `
* `class SQLOpenHashSet[@specialized(Long, Int, Double, Float) T: ClassTag](`
* `public class ColumnIOUtil `
* `public class OrcColumnStatistics `
* `public class OrcFooterReader `
* `case class OptimizeSkewedJoin(ensureRequirements: EnsureRequirements)`
* `case class SkewJoinChildWrapper(plan: SparkPlan) extends LeafExecNode `
* `case class SimpleCostEvaluator(forceOptimizeSkewedJoin: Boolean) extends CostEvaluator `
* `case class SetCatalogCommand(catalogName: String) extends LeafRunnableCommand `
* `case class SetNamespaceCommand(namespace: Seq[String]) extends LeafRunnableCommand `
* `case class ShowCatalogsCommand(pattern: Option[String]) extends LeafRunnableCommand `
* `case class ShowCurrentNamespaceCommand() extends LeafRunnableCommand `
* `case class WriterBucketSpec(`
* `case class ParquetColumn(`
* `case class CreateIndexExec(`
* `case class DropIndexExec(`
* `case class PushedDownOperators(`
* `case class TableSampleInfo(`
* `case class EnsureRequirements(`
* `trait MapInBatchExec extends UnaryExecNode `
* `case class PythonMapInArrowExec(`
* `class AvailableNowDataStreamWrapper(val delegate: SparkDataStream)`
* `class AvailableNowMicroBatchStreamWrapper(delegate: MicroBatchStream)`
* `class AvailableNowSourceWrapper(delegate: Source)`
* `case class SingleBatchExecutor() extends TriggerExecutor `
* `case class MultiBatchExecutor() extends TriggerExecutor `
* `class RatePerMicroBatchProvider extends SimpleTableProvider with DataSourceRegister `
* `class RatePerMicroBatchTable(`
* `class RatePerMicroBatchStream(`
* `case class RatePerMicroBatchStreamOffset(offset: Long, timestamp: Long) extends Offset `
* `case class RatePerMicroBatchStreamInputPartition(`
* `class RatePerMicroBatchStreamPartitionReader(`
* ` // When this is enabled, this class does additional lookup on write operations (put/delete) to`
* `class SparkUDFExpressionBuilder extends FunctionExpressionBuilder `
* `class HiveThriftServer2AppStatusStore(store: KVStore) `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-985888716
**[Test build #145918 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145918/testReport)** for PR 32031 at commit [`2300dc4`](https://github.com/apache/spark/commit/2300dc405206e053e38ea2ccdb5936969c2271ef).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32031:
URL: https://github.com/apache/spark/pull/32031#issuecomment-837717297
**[Test build #138353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138353/testReport)** for PR 32031 at commit [`48a0be7`](https://github.com/apache/spark/commit/48a0be7b58a1a9d5c556b35359427fb0cd903b13).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #32031: [WIP] Initial work of Remote Shuffle Service on Kubernetes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #32031:
URL: https://github.com/apache/spark/pull/32031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org