You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "SteNicholas (via GitHub)" <gi...@apache.org> on 2024/12/19 08:58:21 UTC

[PR] [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation [celeborn]

SteNicholas opened a new pull request, #3012:
URL: https://github.com/apache/celeborn/pull/3012

   ### What changes were proposed in this pull request?
   
    Support `ShuffleFallbackCount` metric for fallback to vanilla Flink built-in shuffle implementation.
   
   ### Why are the changes needed?
   
   #2932 has already supported fallback to vanilla Flink built-in shuffle implementation, which is lack of `ShuffleFallbackCount` metric to feedback the situation of fallback.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3012:
URL: https://github.com/apache/celeborn/pull/3012#discussion_r1891892222


##########
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/fallback/ShuffleFallbackPolicyRunner.java:
##########
@@ -32,7 +32,7 @@ public class ShuffleFallbackPolicyRunner {
   private static final List<ShuffleFallbackPolicy> FALLBACK_POLICIES =
       ShuffleFallbackPolicyFactory.getShuffleFallbackPolicies();
 
-  public static boolean applyFallbackPolicies(
+  public static Optional<ShuffleFallbackPolicy> applyFallbackPolicies(

Review Comment:
   I think this function name can be changed into `getActiveFallbackPolicy` to match its logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on code in PR #3012:
URL: https://github.com/apache/celeborn/pull/3012#discussion_r1891892222


##########
client-flink/common/src/main/java/org/apache/celeborn/plugin/flink/fallback/ShuffleFallbackPolicyRunner.java:
##########
@@ -32,7 +32,7 @@ public class ShuffleFallbackPolicyRunner {
   private static final List<ShuffleFallbackPolicy> FALLBACK_POLICIES =
       ShuffleFallbackPolicyFactory.getShuffleFallbackPolicies();
 
-  public static boolean applyFallbackPolicies(
+  public static Optional<ShuffleFallbackPolicy> applyFallbackPolicies(

Review Comment:
   I think this function name can be changed into `getActivatedFallbackPolicy` to match its logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX closed pull request #3012: [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation
URL: https://github.com/apache/celeborn/pull/3012


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3012:
URL: https://github.com/apache/celeborn/pull/3012#issuecomment-2553331808

   Ping @reswqa, @codenohup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation [celeborn]

Posted by "codenohup (via GitHub)" <gi...@apache.org>.
codenohup commented on PR #3012:
URL: https://github.com/apache/celeborn/pull/3012#issuecomment-2553488532

   Hi, @SteNicholas 
   
   This PR LGTM, but I have a minor question. It seems that the decision to fall back to the built-in shuffle is based on `job` granularity rather than `Shuffle` granularity. I think it would be beneficial to have a fallback count metric that reflects `job` granularity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-1700][FOLLOWUP] Support ShuffleFallbackCount metric for fallback to vanilla Flink built-in shuffle implementation [celeborn]

Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #3012:
URL: https://github.com/apache/celeborn/pull/3012#issuecomment-2553508997

   @codenohup, The difference of implementation that Flink only supports job granular fallback but Spark supports shuffle granular fallback causes above fallback metric situation. I would like to support job granular fallback metrics in another pull requests 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org