You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2024/01/10 02:52:32 UTC
[PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
zuston opened a new pull request, #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429
### What changes were proposed in this pull request?
allow specifying negative fallback threshold to avoid event being discarded
### Why are the changes needed?
Fix: #1428
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1888443945
I can't get the point of this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453248016
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
Review Comment:
Why do we remove this check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1891338475
cc @xianjingfeng If you have time, could you help review this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1888326217
PTAL @leixm @jerqi . I think this is a critical bug in 0.8 branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1894980164
Merged. Thanks @xianjingfeng @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1452461881
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,9 +352,7 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
- ConfigUtils.NON_NEGATIVE_LONG_VALIDATOR, " fallback times must be non-negative")
- .defaultValue(0L)
+ .defaultValue(-1L)
Review Comment:
Ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453520611
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
Review Comment:
What about the behaviour of `0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1884318744
## [Codecov](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1429?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`5d027c0`)](https://app.codecov.io/gh/apache/incubator-uniffle/commit/5d027c0ccc4f1c24225e5639f0a44142d4b7b1ae?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 53.38% compared to head [(`cefbcd7`)](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1429?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) 54.25%.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #1429 +/- ##
============================================
+ Coverage 53.38% 54.25% +0.87%
+ Complexity 2729 2726 -3
============================================
Files 422 402 -20
Lines 24046 21684 -2362
Branches 2051 2051
============================================
- Hits 12836 11764 -1072
+ Misses 10412 9193 -1219
+ Partials 798 727 -71
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/incubator-uniffle/pull/1429?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "xianjingfeng (via GitHub)" <gi...@apache.org>.
xianjingfeng commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1452037189
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,9 +352,7 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
- ConfigUtils.NON_NEGATIVE_LONG_VALIDATOR, " fallback times must be non-negative")
- .defaultValue(0L)
+ .defaultValue(-1L)
Review Comment:
1. What if the users set it to 0.
2. If `!storageManager.canWrite(flushEvent)`, we should ignore retry times. So should we add a new argument named `force` to `org.apache.uniffle.server.storage.AbstractStorageManagerFallbackStrategy#tryFallback` for ignoring retry times. WDYT? @zuston @jerqi
https://github.com/apache/incubator-uniffle/blob/318050144b62c0256c151a278bb04ac73c6e544b/server/src/main/java/org/apache/uniffle/server/storage/hybrid/FallbackBasedStorageManagerSelector.java#L49-L51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1893437997
PTAL again @xianjingfeng
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1452039966
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,9 +352,7 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
- ConfigUtils.NON_NEGATIVE_LONG_VALIDATOR, " fallback times must be non-negative")
- .defaultValue(0L)
+ .defaultValue(-1L)
Review Comment:
SGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1454348905
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
Review Comment:
For the first time fallback, it will reject for 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#issuecomment-1888446934
> I can't get the point of this pr.
Please see the test case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): fallback invalid when local storage can't write [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453381090
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
Review Comment:
if -1,it means we could fallback directly once failed at the first time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1428] fix(server): allow specifying negative fallback threshold to avoid event being discarded [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on code in PR #1429:
URL: https://github.com/apache/incubator-uniffle/pull/1429#discussion_r1453167972
##########
server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java:
##########
@@ -352,8 +352,6 @@ public class ShuffleServerConf extends RssBaseConf {
public static final ConfigOption<Long> FALLBACK_MAX_FAIL_TIMES =
ConfigOptions.key("rss.server.hybrid.storage.fallback.max.fail.times")
.longType()
- .checkValue(
Review Comment:
The negative check is unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org