You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "EnricoMi (via GitHub)" <gi...@apache.org> on 2024/03/05 12:19:08 UTC

[PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

EnricoMi opened a new pull request, #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557

   ### What changes were proposed in this pull request?
   Fetches dynamic client config as early as possible, to be able to use dynamic client config to create the shuffle client with updated config.
   
   ### Why are the changes needed?
   Providing config or the shuffle client via coordinator is operationally useful as cluster-wide settings can be deployed through the cluster and changed over time. Clients and apps do not need to change configs.
   
   Fixes Spark part of #1554
   
   ### Does this PR introduce _any_ user-facing change?
   More configs can be provided via coordinators.
   
   ### How was this patch tested?
   Existing and [follow-up](https://github.com/apache/incubator-uniffle/pull/1528/files#diff-ea644edb1c0bf0e80f9a960adbc1615c99cb6a3a0d5fe24f788307f1daf22f46R127-R131) unit tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513907449


##########
client-spark/spark3/src/test/java/org/apache/spark/shuffle/DelegationRssShuffleManagerTest.java:
##########
@@ -78,6 +78,7 @@ public void testCreateFallback() throws Exception {
     conf.set(RssSparkConfig.RSS_DYNAMIC_CLIENT_CONF_ENABLED.key(), "false");
     conf.set(RssSparkConfig.RSS_ACCESS_ID.key(), "mockId");
     conf.set(RssSparkConfig.RSS_ENABLED.key(), "true");
+    conf.set(RssSparkConfig.RSS_STORAGE_TYPE.key(), "MEMORY_LOCALFILE");

Review Comment:
   Coordinator can apply the storage type to the client. Will this pull request affect this feature.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#issuecomment-1980004861

   And I think the PR title is not proper for this change, which don't reflect the feature of overwriting the existing client conf.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1522399624


##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -214,16 +221,6 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
                     .unregisterRequestTimeSec(unregisterRequestTimeoutSec)
                     .rssConf(rssConf));
     registerCoordinator();
-    // fetch client conf and apply them if necessary and disable ESS
-    if (isDriver && dynamicConfEnabled) {
-      Map<String, String> clusterClientConf =
-          shuffleWriteClient.fetchClientConf(

Review Comment:
   Got it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513893667


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkShuffleUtils.java:
##########
@@ -130,9 +130,13 @@ public static void applyDynamicClientConf(SparkConf sparkConf, Map<String, Strin
         sparkConfKey = RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + sparkConfKey;
       }
       String confVal = kv.getValue();
-      if (!sparkConf.contains(sparkConfKey)
-          || RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey)) {
-        LOG.warn("Use conf dynamic conf {} = {}", sparkConfKey, confVal);
+      boolean isMandatory = RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey);

Review Comment:
   Oh I misunderstand this. I think you just want to re-create client after fetching client conf. Right? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#issuecomment-1980206649

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1557?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 53.78%. Comparing base [(`9f4ccde`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/9f4ccde36bcce22e033dc7c7b83f263e6e252f53?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`b687162`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1557?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 1 commits behind head on master.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1557      +/-   ##
   ============================================
   + Coverage     53.63%   53.78%   +0.15%     
   + Complexity     2822     2269     -553     
   ============================================
     Files           436      358      -78     
     Lines         24549    15920    -8629     
     Branches       2080     1447     -633     
   ============================================
   - Hits          13167     8563    -4604     
   + Misses        10562     6869    -3693     
   + Partials        820      488     -332     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1557?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513889174


##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -278,6 +280,33 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
     this.reassignedFaultyServers = JavaUtils.newConcurrentMap();
   }
 
+  private static void fetchAndApplyDynamicConf(SparkConf sparkConf) {
+    String clientType = sparkConf.get(RssSparkConfig.RSS_CLIENT_TYPE);
+    String coordinators = sparkConf.get(RssSparkConfig.RSS_COORDINATOR_QUORUM.key());
+    CoordinatorClientFactory coordinatorClientFactory = CoordinatorClientFactory.getInstance();
+    List<CoordinatorClient> coordinatorClients =
+        coordinatorClientFactory.createCoordinatorClient(
+            ClientType.valueOf(clientType), coordinators);
+
+    int timeoutMs =
+        sparkConf.getInt(
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.key(),
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.defaultValue().get());
+    for (CoordinatorClient client : coordinatorClients) {
+      RssFetchClientConfResponse response =

Review Comment:
   Using `shuffleWriteClient.fetchClientConf` to fetch the dynamic client conf disallows configuring the creation of `ShuffleWriteClient` through dynamic config. See #1554.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1515908428


##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -214,16 +221,6 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
                     .unregisterRequestTimeSec(unregisterRequestTimeoutSec)
                     .rssConf(rssConf));
     registerCoordinator();
-    // fetch client conf and apply them if necessary and disable ESS
-    if (isDriver && dynamicConfEnabled) {
-      Map<String, String> clusterClientConf =
-          shuffleWriteClient.fetchClientConf(

Review Comment:
   This method is still used by MR and TEZ clients.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#issuecomment-1978706714

   ## Test Results
   2 312 files  ±0  2 312 suites  ±0   4h 36m 46s :stopwatch: + 1m 10s
     823 tests ±0    822 :white_check_mark: ±0   1 :zzz: ±0  0 :x: ±0 
   9 697 runs  ±0  9 683 :white_check_mark: ±0  14 :zzz: ±0  0 :x: ±0 
   
   Results for commit 7f9904eb. ± Comparison against base commit d8aedf34.
   
   [test-results]:data:application/gzip;base64,H4sIAK8U52UC/03MTQ6DIBRG0a0Yxh3wr3QzjcAjIVVpEEamey8aoQy/w+MeyPkFdvQcKCP0MaA9+9Rvm+OcfNiKECmxLFTe03kxUVbXa8/GXET/9Paf81cDN/ulAG4AMYZ4S8zb2VRSjfeqTSUn1uhu8gpd89p90oR19akMNDqlMAfNMQFHZm40lVRMDAA7oTXRwlriDKDvDzG2vuEQAQAA
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#issuecomment-1994085447

   Fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513884536


##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -159,6 +164,13 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
     }
     this.user = sparkConf.get("spark.rss.quota.user", "user");
     this.uuid = sparkConf.get("spark.rss.quota.uuid", Long.toString(System.currentTimeMillis()));
+
+    // fetch client conf and apply them if necessary
+    if (isDriver && sparkConf.get(RssSparkConfig.RSS_DYNAMIC_CLIENT_CONF_ENABLED)) {

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513725794


##########
client-spark/spark3/src/test/java/org/apache/spark/shuffle/DelegationRssShuffleManagerTest.java:
##########
@@ -78,6 +78,7 @@ public void testCreateFallback() throws Exception {
     conf.set(RssSparkConfig.RSS_DYNAMIC_CLIENT_CONF_ENABLED.key(), "false");
     conf.set(RssSparkConfig.RSS_ACCESS_ID.key(), "mockId");
     conf.set(RssSparkConfig.RSS_ENABLED.key(), "true");
+    conf.set(RssSparkConfig.RSS_STORAGE_TYPE.key(), "MEMORY_LOCALFILE");

Review Comment:
   Why do we need to add this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513906009


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkShuffleUtils.java:
##########
@@ -130,9 +130,13 @@ public static void applyDynamicClientConf(SparkConf sparkConf, Map<String, Strin
         sparkConfKey = RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + sparkConfKey;
       }
       String confVal = kv.getValue();
-      if (!sparkConf.contains(sparkConfKey)
-          || RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey)) {
-        LOG.warn("Use conf dynamic conf {} = {}", sparkConfKey, confVal);
+      boolean isMandatory = RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey);

Review Comment:
   Re-creating the client is one of the suggested solutions: #1554
   
   I thought bootstrapping the config from the coordinator clients first thing would be better design. Re-creating the client looks a bit hacky.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513924833


##########
client-spark/spark3/src/test/java/org/apache/spark/shuffle/DelegationRssShuffleManagerTest.java:
##########
@@ -78,6 +78,7 @@ public void testCreateFallback() throws Exception {
     conf.set(RssSparkConfig.RSS_DYNAMIC_CLIENT_CONF_ENABLED.key(), "false");
     conf.set(RssSparkConfig.RSS_ACCESS_ID.key(), "mockId");
     conf.set(RssSparkConfig.RSS_ENABLED.key(), "true");
+    conf.set(RssSparkConfig.RSS_STORAGE_TYPE.key(), "MEMORY_LOCALFILE");

Review Comment:
   This test calls into `new RssShuffleManager(sparkConf, false)`, which mimics creating the shuffle manager on an executor, and executors do not fetch dynamic config, so the given `sparkConf` must have the storage type set for the test.
   
   The logic of applying config is unchanged, this only happens earlier. I presume there are tests for this use case, which should check the desired behaviour is preserved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513881437


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkShuffleUtils.java:
##########
@@ -130,9 +130,13 @@ public static void applyDynamicClientConf(SparkConf sparkConf, Map<String, Strin
         sparkConfKey = RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + sparkConfKey;
       }
       String confVal = kv.getValue();
-      if (!sparkConf.contains(sparkConfKey)
-          || RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey)) {
-        LOG.warn("Use conf dynamic conf {} = {}", sparkConfKey, confVal);
+      boolean isMandatory = RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey);

Review Comment:
   This is existing, un-changed logic. This only improves logging distinguishing which conf is overridden and which is added due to dynamic conf. Before, both were logged with the same message on WARN level.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513998476


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkShuffleUtils.java:
##########
@@ -130,9 +130,13 @@ public static void applyDynamicClientConf(SparkConf sparkConf, Map<String, Strin
         sparkConfKey = RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + sparkConfKey;
       }
       String confVal = kv.getValue();
-      if (!sparkConf.contains(sparkConfKey)
-          || RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey)) {
-        LOG.warn("Use conf dynamic conf {} = {}", sparkConfKey, confVal);
+      boolean isMandatory = RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey);

Review Comment:
   Got it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513892008


##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -278,6 +280,33 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
     this.reassignedFaultyServers = JavaUtils.newConcurrentMap();
   }
 
+  private static void fetchAndApplyDynamicConf(SparkConf sparkConf) {
+    String clientType = sparkConf.get(RssSparkConfig.RSS_CLIENT_TYPE);
+    String coordinators = sparkConf.get(RssSparkConfig.RSS_COORDINATOR_QUORUM.key());
+    CoordinatorClientFactory coordinatorClientFactory = CoordinatorClientFactory.getInstance();
+    List<CoordinatorClient> coordinatorClients =
+        coordinatorClientFactory.createCoordinatorClient(
+            ClientType.valueOf(clientType), coordinators);
+
+    int timeoutMs =
+        sparkConf.getInt(
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.key(),
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.defaultValue().get());
+    for (CoordinatorClient client : coordinatorClients) {
+      RssFetchClientConfResponse response =

Review Comment:
   Got it. But I want to know why re-creating the write client? Can you share some problems that you encountered? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513762225


##########
client-spark/common/src/main/java/org/apache/spark/shuffle/RssSparkShuffleUtils.java:
##########
@@ -130,9 +130,13 @@ public static void applyDynamicClientConf(SparkConf sparkConf, Map<String, Strin
         sparkConfKey = RssSparkConfig.SPARK_RSS_CONFIG_PREFIX + sparkConfKey;
       }
       String confVal = kv.getValue();
-      if (!sparkConf.contains(sparkConfKey)
-          || RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey)) {
-        LOG.warn("Use conf dynamic conf {} = {}", sparkConfKey, confVal);
+      boolean isMandatory = RssSparkConfig.RSS_MANDATORY_CLUSTER_CONF.contains(sparkConfKey);

Review Comment:
   Please add some test cases for this. 
   
   BTW, it looks the RSS_MANDATORY_CLUSTER_CONF also should be allowed to configure by conf.



##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -159,6 +164,13 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
     }
     this.user = sparkConf.get("spark.rss.quota.user", "user");
     this.uuid = sparkConf.get("spark.rss.quota.uuid", Long.toString(System.currentTimeMillis()));
+
+    // fetch client conf and apply them if necessary
+    if (isDriver && sparkConf.get(RssSparkConfig.RSS_DYNAMIC_CLIENT_CONF_ENABLED)) {

Review Comment:
   why not using `dynamicConfEnabled`



##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -278,6 +280,33 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
     this.reassignedFaultyServers = JavaUtils.newConcurrentMap();
   }
 
+  private static void fetchAndApplyDynamicConf(SparkConf sparkConf) {
+    String clientType = sparkConf.get(RssSparkConfig.RSS_CLIENT_TYPE);
+    String coordinators = sparkConf.get(RssSparkConfig.RSS_COORDINATOR_QUORUM.key());
+    CoordinatorClientFactory coordinatorClientFactory = CoordinatorClientFactory.getInstance();
+    List<CoordinatorClient> coordinatorClients =
+        coordinatorClientFactory.createCoordinatorClient(
+            ClientType.valueOf(clientType), coordinators);
+
+    int timeoutMs =
+        sparkConf.getInt(
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.key(),
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.defaultValue().get());
+    for (CoordinatorClient client : coordinatorClients) {
+      RssFetchClientConfResponse response =

Review Comment:
   why not using `shuffleWriteClient.fetchClientConf` ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1515671804


##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -214,16 +221,6 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
                     .unregisterRequestTimeSec(unregisterRequestTimeoutSec)
                     .rssConf(rssConf));
     registerCoordinator();
-    // fetch client conf and apply them if necessary and disable ESS
-    if (isDriver && dynamicConfEnabled) {
-      Map<String, String> clusterClientConf =
-          shuffleWriteClient.fetchClientConf(

Review Comment:
   Can you delete the `fetchClientConf` method in `shuffleWriteClient` ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513904380


##########
client-spark/spark3/src/test/java/org/apache/spark/shuffle/DelegationRssShuffleManagerTest.java:
##########
@@ -78,6 +78,7 @@ public void testCreateFallback() throws Exception {
     conf.set(RssSparkConfig.RSS_DYNAMIC_CLIENT_CONF_ENABLED.key(), "false");
     conf.set(RssSparkConfig.RSS_ACCESS_ID.key(), "mockId");
     conf.set(RssSparkConfig.RSS_ENABLED.key(), "true");
+    conf.set(RssSparkConfig.RSS_STORAGE_TYPE.key(), "MEMORY_LOCALFILE");

Review Comment:
   This test further down asserts an exception is thrown. Moving `RssSparkShuffleUtils.validateRssClientConf(sparkConf)` up in the `RssShuffleManager` constructor throws a different exception if the storage type is not set.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on code in PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#discussion_r1513926214


##########
client-spark/spark3/src/main/java/org/apache/spark/shuffle/RssShuffleManager.java:
##########
@@ -278,6 +280,33 @@ public RssShuffleManager(SparkConf conf, boolean isDriver) {
     this.reassignedFaultyServers = JavaUtils.newConcurrentMap();
   }
 
+  private static void fetchAndApplyDynamicConf(SparkConf sparkConf) {
+    String clientType = sparkConf.get(RssSparkConfig.RSS_CLIENT_TYPE);
+    String coordinators = sparkConf.get(RssSparkConfig.RSS_COORDINATOR_QUORUM.key());
+    CoordinatorClientFactory coordinatorClientFactory = CoordinatorClientFactory.getInstance();
+    List<CoordinatorClient> coordinatorClients =
+        coordinatorClientFactory.createCoordinatorClient(
+            ClientType.valueOf(clientType), coordinators);
+
+    int timeoutMs =
+        sparkConf.getInt(
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.key(),
+            RssSparkConfig.RSS_ACCESS_TIMEOUT_MS.defaultValue().get());
+    for (CoordinatorClient client : coordinatorClients) {
+      RssFetchClientConfResponse response =

Review Comment:
   Configuring the block id layout via dynamic conf requires this: https://github.com/apache/incubator-uniffle/pull/1528/files#diff-ea644edb1c0bf0e80f9a960adbc1615c99cb6a3a0d5fe24f788307f1daf22f46R127-R128



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "EnricoMi (via GitHub)" <gi...@apache.org>.
EnricoMi commented on PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#issuecomment-1980208347

   > ... overwriting the existing client conf.
   
   this is not new, this existed before


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1554] feat(spark): Fetch dynamic client conf as early as possible [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1557:
URL: https://github.com/apache/incubator-uniffle/pull/1557#issuecomment-1993188130

   Could you help resolve the conflict? @EnricoMi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org