You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2024/01/10 02:52:32 UTC

[PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

zuston opened a new pull request, #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429

   ### What changes were proposed in this pull request?
   
   allow specifying negative fallback threshold to avoid event being discarded
   
   ### Why are the changes needed?
   
   Fix: #1428 
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Unit tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1888443945

   I can't get the point of this pr.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453248016


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(

Review Comment:
   Why do we remove this check?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1891338475

   cc @xianjingfeng If you have time, could you help review this ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1888326217

   PTAL @leixm @jerqi . I think this is a critical bug in 0.8 branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1894980164

   Merged. Thanks @xianjingfeng @jerqi 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1452461881


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,9 +352,7 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(
-              ConfigUtils.NON_NEGATIVE_LONG_VALIDATOR, " fallback times must be non-negative")
-          .defaultValue(0L)
+          .defaultValue(-1L)

Review Comment:
   Ok for me. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]

Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453520611


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(

Review Comment:
   What about the behaviour of `0`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1884318744

   ## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1429?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Comparison is base [(`5d027c0`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/5d027c0ccc4f1c24225e5639f0a44142d4b7b1ae?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.38% compared to head [(`cefbcd7`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1429?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.25%.
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #1429      +/-   ##
   ============================================
   + Coverage     53.38%   54.25%   +0.87%     
   + Complexity     2729     2726       -3     
   ============================================
     Files           422      402      -20     
     Lines         24046    21684    -2362     
     Branches       2051     2051              
   ============================================
   - Hits          12836    11764    -1072     
   + Misses        10412     9193    -1219     
   + Partials        798      727      -71     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1429?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1452037189


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,9 +352,7 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(
-              ConfigUtils.NON_NEGATIVE_LONG_VALIDATOR, " fallback times must be non-negative")
-          .defaultValue(0L)
+          .defaultValue(-1L)

Review Comment:
   1. What if the users set it to 0.
   2. If `!storageManager.canWrite(flushEvent)`, we should ignore retry times. So should we add a new argument named `force` to `org.apache.uniffle.server.storage.AbstractStorageManagerFallbackStrategy#tryFallback` for ignoring retry times. WDYT? @zuston  @jerqi 
   
   https://github.com/apache/incubator-uniffle/blob/318050144b62c0256c151a278bb04ac73c6e544b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/FallbackBasedStorageManagerSelector.java#L49-L51



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1893437997

   PTAL again @xianjingfeng 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1452039966


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,9 +352,7 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(
-              ConfigUtils.NON_NEGATIVE_LONG_VALIDATOR, " fallback times must be non-negative")
-          .defaultValue(0L)
+          .defaultValue(-1L)

Review Comment:
   SGTM. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1454348905


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(

Review Comment:
   For the first time fallback, it will reject for 0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1888446934

   > I can't get the point of this pr.
   
   Please see the test case


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453381090


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(

Review Comment:
   if -1,it means we could fallback directly once failed at the first time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org


Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]

Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453167972


##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
   public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
       ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
           .longType()
-          .checkValue(

Review Comment:
   The negative check is unnecessary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org