You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2024/05/07 09:50:43 UTC
[PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
zuston opened a new pull request, #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677
### What changes were proposed in this pull request?
Support report blockIds from shuffle-servers to spark driver optionally
### Why are the changes needed?
Fix: #1538
### Does this PR introduce _any_ user-facing change?
Yes. `rss.client.blockId.selfManagedEnabled` is introduced, default value is false.
### How was this patch tested?
Integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596200748
##########
integration-test/spark3/src/test/java/org/apache/uniffle/test/ContinuousSelectPartitionStrategyTest.java:
##########
@@ -238,8 +239,12 @@ public int compare(String o1, String o2) {
// Validate getShuffleResultForMultiPart is correct before return result
ClientType clientType =
ClientType.valueOf(spark.sparkContext().getConf().get(RssSparkConfig.RSS_CLIENT_TYPE));
- if (ClientType.GRPC == clientType) {
+ boolean blockIdSelfManagedEnabled =
+ RssSparkConfig.toRssConf(spark.sparkContext().getConf())
+ .get(RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED);
+ if (ClientType.GRPC == clientType && !blockIdSelfManagedEnabled) {
// TODO skip validating for GRPC_NETTY, needs to mock ShuffleServerNettyHandler
+ // TODO: skip validating when blockId is managed in spark driver side.
Review Comment:
Why is this a `TODO`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1593324321
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/BlockIdManager.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.shuffle;
+
+import java.util.List;
+import java.util.Map;
+
+import org.roaringbitmap.longlong.Roaring64NavigableMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.uniffle.common.util.JavaUtils;
+import org.apache.uniffle.common.util.RssUtils;
+
+/** The class is to manage the shuffle data blockIds in spark driver side. */
+public class BlockIdManager {
Review Comment:
Yes, I will try this and apply this into production to check the performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596206339
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkConfig.java:
##########
@@ -37,6 +37,12 @@
public class RssSparkConfig {
+ public static final ConfigOption<Boolean> RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED =
+ ConfigOptions.key("rss.client.blockId.selfManagedEnabled")
Review Comment:
selfmanaged is adjective. Here the noun is more proper. π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596201296
##########
integration-test/spark3/src/test/java/org/apache/uniffle/test/ContinuousSelectPartitionStrategyTest.java:
##########
@@ -238,8 +239,12 @@ public int compare(String o1, String o2) {
// Validate getShuffleResultForMultiPart is correct before return result
ClientType clientType =
ClientType.valueOf(spark.sparkContext().getConf().get(RssSparkConfig.RSS_CLIENT_TYPE));
- if (ClientType.GRPC == clientType) {
+ boolean blockIdSelfManagedEnabled =
+ RssSparkConfig.toRssConf(spark.sparkContext().getConf())
+ .get(RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED);
+ if (ClientType.GRPC == clientType && !blockIdSelfManagedEnabled) {
// TODO skip validating for GRPC_NETTY, needs to mock ShuffleServerNettyHandler
+ // TODO: skip validating when blockId is managed in spark driver side.
Review Comment:
Nice catch. todo could be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#issuecomment-2105522367
> Wait for the performance report.
I think I will merge this. When this feature is appied in our internal cluster, the performance report will be attached.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596205553
##########
client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -810,25 +823,31 @@ private ShuffleManagerClient createShuffleManagerClient(String host, int port) {
.createShuffleManagerClient(ClientType.GRPC, host, port);
}
+ private ShuffleManagerClient getOrCreateShuffleManagerClient() {
Review Comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596217907
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkConfig.java:
##########
@@ -37,6 +37,12 @@
public class RssSparkConfig {
+ public static final ConfigOption<Boolean> RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED =
+ ConfigOptions.key("rss.client.blockId.selfManagedEnabled")
Review Comment:
> Maybe we should remove client word to align other blockId config options.
Done
> Except, I'm a little worried about the config options about Uniffle. Because there are some less elegant issues. client seems extra. Don't use camel style. Proper segments.
This could be optimized because we have the discard mechanism in configOption, that need to be detailed in doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596213804
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkConfig.java:
##########
@@ -37,6 +37,12 @@
public class RssSparkConfig {
+ public static final ConfigOption<Boolean> RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED =
+ ConfigOptions.key("rss.client.blockId.selfManagedEnabled")
Review Comment:
Maybe we should remove `client` word to align other config options.
Except, I'm a little worried about the config options about Uniffle. Because there are some less elegant issues. `client` seems extra. Don't use camel style. Proper segments.
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkConfig.java:
##########
@@ -37,6 +37,12 @@
public class RssSparkConfig {
+ public static final ConfigOption<Boolean> RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED =
+ ConfigOptions.key("rss.client.blockId.selfManagedEnabled")
Review Comment:
Maybe we should remove `client` word to align other `blockId` config options.
Except, I'm a little worried about the config options about Uniffle. Because there are some less elegant issues. `client` seems extra. Don't use camel style. Proper segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#issuecomment-2097967685
## Test Results
β2β377 filesβ β-β7βββ2β377 suitesβ β-β7βββ5h 2m 12s :stopwatch: + 23m 26s
βββ922 tests Β±0βββββ916 :white_check_mark: β-ββ5βββ1 :zzz: Β±0βββ2 :x: +β2βββ3 :fire: +β3β
10β684 runsβ β-β7ββ10β644 :white_check_mark: β-β33ββ14 :zzz: Β±0ββ12 :x: +12ββ14 :fire: +14β
For more details on these failures and errors, see [this check](https://github.com/apache/incubator-uniffle/runs/24673867504).
Results for commit 01b8c567.βΒ± Comparison against base commit 6d193278.
[test-results]:data:application/gzip;base64,H4sIAPD/OWYC/03MTQ7CIBCG4as0rF0wQAt4GTNSSIj9MRRWjXd3MC11+T0M785CnPzG7p2QWt86tpWY//dYEua4LiRgQAoies/1woq2HltxrhIMF73iu/5qEDBONdzAp7QmEkmSylKbwAejjnlGydRlR7XBUQVxwln9nbh1nmOmxTg8jesHzZGDDR4V9lKAMYq7UYUeR6u91QbZ5ws5yRhLFAEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1593302296
##########
client-spark/common/src/main/java/org/apache/uniffle/shuffle/BlockIdManager.java:
##########
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.uniffle.shuffle;
+
+import java.util.List;
+import java.util.Map;
+
+import org.roaringbitmap.longlong.Roaring64NavigableMap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.uniffle.common.util.JavaUtils;
+import org.apache.uniffle.common.util.RssUtils;
+
+/** The class is to manage the shuffle data blockIds in spark driver side. */
+public class BlockIdManager {
Review Comment:
I have a little concern about performance. We ever stored all block ids in one shuffle server. It has poor performance. Because bitmap has poor performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#issuecomment-2103787051
You need to add a document for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596201890
##########
client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -810,25 +823,31 @@ private ShuffleManagerClient createShuffleManagerClient(String host, int port) {
.createShuffleManagerClient(ClientType.GRPC, host, port);
}
+ private ShuffleManagerClient getOrCreateShuffleManagerClient() {
Review Comment:
Will this be used for multiple threads?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#issuecomment-2103812531
> LGTM, wait for your performance report.
This should wait some time π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596213804
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkConfig.java:
##########
@@ -37,6 +37,12 @@
public class RssSparkConfig {
+ public static final ConfigOption<Boolean> RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED =
+ ConfigOptions.key("rss.client.blockId.selfManagedEnabled")
Review Comment:
Maybe we should remove `client` word to align other config options.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596205383
##########
integration-test/spark3/src/test/java/org/apache/uniffle/test/ContinuousSelectPartitionStrategyTest.java:
##########
@@ -244,7 +244,7 @@ public int compare(String o1, String o2) {
.get(RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED);
if (ClientType.GRPC == clientType && !blockIdSelfManagedEnabled) {
// TODO skip validating for GRPC_NETTY, needs to mock ShuffleServerNettyHandler
- // TODO: skip validating when blockId is managed in spark driver side.
+ // skip validating when blockId is managed in spark driver side.
Review Comment:
Maybe you can remove the `TODO` in the 246 line, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596203273
##########
client-spark/spark2/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -810,25 +823,31 @@ private ShuffleManagerClient createShuffleManagerClient(String host, int port) {
.createShuffleManagerClient(ClientType.GRPC, host, port);
}
+ private ShuffleManagerClient getOrCreateShuffleManagerClient() {
Review Comment:
Won't. The client will be initialized in this class constructor.
##########
integration-test/spark3/src/test/java/org/apache/uniffle/test/ContinuousSelectPartitionStrategyTest.java:
##########
@@ -238,8 +239,12 @@ public int compare(String o1, String o2) {
// Validate getShuffleResultForMultiPart is correct before return result
ClientType clientType =
ClientType.valueOf(spark.sparkContext().getConf().get(RssSparkConfig.RSS_CLIENT_TYPE));
- if (ClientType.GRPC == clientType) {
+ boolean blockIdSelfManagedEnabled =
+ RssSparkConfig.toRssConf(spark.sparkContext().getConf())
+ .get(RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED);
+ if (ClientType.GRPC == clientType && !blockIdSelfManagedEnabled) {
// TODO skip validating for GRPC_NETTY, needs to mock ShuffleServerNettyHandler
+ // TODO: skip validating when blockId is managed in spark driver side.
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#issuecomment-2105461969
Minor update. PTAL @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#issuecomment-2105524180
> > Wait for the performance report.
>
> I think I will merge this. When this feature is appied in our internal cluster, the performance report will be attached.
Ok for me, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#issuecomment-2103788811
> You need to add a document for this.
Yes I will if you have other comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596206496
##########
integration-test/spark3/src/test/java/org/apache/uniffle/test/ContinuousSelectPartitionStrategyTest.java:
##########
@@ -244,7 +244,7 @@ public int compare(String o1, String o2) {
.get(RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED);
if (ClientType.GRPC == clientType && !blockIdSelfManagedEnabled) {
// TODO skip validating for GRPC_NETTY, needs to mock ShuffleServerNettyHandler
- // TODO: skip validating when blockId is managed in spark driver side.
+ // skip validating when blockId is managed in spark driver side.
Review Comment:
No. todo in 246 is reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1538] feat(spark): report blockIds to spark driver optionally [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1677:
URL: https://github.com/apache/incubator-uniffle/pull/1677#discussion_r1596200327
##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkConfig.java:
##########
@@ -37,6 +37,12 @@
public class RssSparkConfig {
+ public static final ConfigOption<Boolean> RSS_CLIENT_BLOCK_ID_SELF_MANAGED_ENABLED =
+ ConfigOptions.key("rss.client.blockId.selfManagedEnabled")
Review Comment:
`selfManagementEnabled`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org