You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "jerqi (via GitHub)" <gi...@apache.org> on 2023/02/22 12:24:52 UTC
[GitHub] [incubator-uniffle] jerqi opened a new pull request, #648: [#647] fix: Multiple coordinator delect the same file
jerqi opened a new pull request, #648:
URL: https://github.com/apache/incubator-uniffle/pull/648
### What changes were proposed in this pull request?
Add coordinator's id to the file which Coordinator delect.
### Why are the changes needed?
Fix #647
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] codecov-commenter commented on pull request #648: [#647] fix: Multiple coordinator delect the same file
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #648:
URL: https://github.com/apache/incubator-uniffle/pull/648#issuecomment-1439945123
# [Codecov](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#648](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (359644d) into [master](https://codecov.io/gh/apache/incubator-uniffle/commit/c45c187b6905b37d93abfc8e0ab2b5829ab18096?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c45c187) will **increase** coverage by `0.85%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #648 +/- ##
============================================
+ Coverage 60.89% 61.74% +0.85%
+ Complexity 1798 1707 -91
============================================
Files 214 193 -21
Lines 12381 9928 -2453
Branches 1042 1010 -32
============================================
- Hits 7539 6130 -1409
+ Misses 4438 3465 -973
+ Partials 404 333 -71
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...che/uniffle/coordinator/util/CoordinatorUtils.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3IvdXRpbC9Db29yZGluYXRvclV0aWxzLmphdmE=) | `72.00% <ø> (ø)` | |
| [.../apache/uniffle/coordinator/CoordinatorServer.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3IvQ29vcmRpbmF0b3JTZXJ2ZXIuamF2YQ==) | `57.37% <100.00%> (+0.35%)` | :arrow_up: |
| [...ategy/storage/AppBalanceSelectStorageStrategy.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3Ivc3RyYXRlZ3kvc3RvcmFnZS9BcHBCYWxhbmNlU2VsZWN0U3RvcmFnZVN0cmF0ZWd5LmphdmE=) | `73.07% <100.00%> (+1.07%)` | :arrow_up: |
| [...orage/LowestIOSampleCostSelectStorageStrategy.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29vcmRpbmF0b3Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3VuaWZmbGUvY29vcmRpbmF0b3Ivc3RyYXRlZ3kvc3RvcmFnZS9Mb3dlc3RJT1NhbXBsZUNvc3RTZWxlY3RTdG9yYWdlU3RyYXRlZ3kuamF2YQ==) | `71.92% <100.00%> (+1.02%)` | :arrow_up: |
| [...a/org/apache/uniffle/server/RegisterHeartBeat.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9SZWdpc3RlckhlYXJ0QmVhdC5qYXZh) | `43.85% <0.00%> (-43.86%)` | :arrow_down: |
| [.../java/org/apache/uniffle/server/ShuffleServer.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9TaHVmZmxlU2VydmVyLmphdmE=) | `62.06% <0.00%> (-2.32%)` | :arrow_down: |
| [...he/uniffle/server/buffer/ShuffleBufferManager.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2VydmVyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL3NlcnZlci9idWZmZXIvU2h1ZmZsZUJ1ZmZlck1hbmFnZXIuamF2YQ==) | `82.74% <0.00%> (-0.36%)` | :arrow_down: |
| [.../java/org/apache/uniffle/common/util/RssUtils.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NvbW1vbi91dGlsL1Jzc1V0aWxzLmphdmE=) | `59.31% <0.00%> (ø)` | |
| [.../org/apache/uniffle/common/config/ConfigUtils.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NvbW1vbi9jb25maWcvQ29uZmlnVXRpbHMuamF2YQ==) | `95.55% <0.00%> (ø)` | |
| [...rg/apache/uniffle/common/config/ConfigOptions.java](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-Y29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS91bmlmZmxlL2NvbW1vbi9jb25maWcvQ29uZmlnT3B0aW9ucy5qYXZh) | `96.61% <0.00%> (ø)` | |
| ... and [28 more](https://codecov.io/gh/apache/incubator-uniffle/pull/648?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #648: [#647] fix: Multiple coordinator produce conflicts when they delect the same file
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #648:
URL: https://github.com/apache/incubator-uniffle/pull/648#issuecomment-1441120075
This is a fix, I will merge this pr to branch-0.7, too. cc @zuston
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] jerqi commented on pull request #648: [#647] fix: Multiple coordinator produce conflicts when they delect the same file
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #648:
URL: https://github.com/apache/incubator-uniffle/pull/648#issuecomment-1440139447
addressed comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on pull request #648: [#647] fix: Multiple coordinator produce conflicts when they delect the same file
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #648:
URL: https://github.com/apache/incubator-uniffle/pull/648#issuecomment-1441114660
Thanks, merged this into master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy commented on a diff in pull request #648: [#647] fix: Multiple coordinator produce conflicts when they delect the same file
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #648:
URL: https://github.com/apache/incubator-uniffle/pull/648#discussion_r1114298822
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/strategy/storage/LowestIOSampleCostSelectStorageStrategy.java:
##########
@@ -64,6 +66,8 @@ public LowestIOSampleCostSelectStorageStrategy(
this.availableRemoteStorageInfo = availableRemoteStorageInfo;
this.hdfsConf = new Configuration();
readAndWriteTimes = conf.getInteger(CoordinatorConf.COORDINATOR_REMOTE_STORAGE_SCHEDULE_ACCESS_TIMES);
+ coordinatorId = conf.getString(CoordinatorUtils.COORDINATOR_ID,
+ String.valueOf(CoordinatorUtils.getRandomInt()));
Review Comment:
Usually an UUID should be used to avoid duplication.
##########
coordinator/src/main/java/org/apache/uniffle/coordinator/strategy/storage/AppBalanceSelectStorageStrategy.java:
##########
@@ -132,4 +136,9 @@ public synchronized RemoteStorageInfo pickStorage(String appId) {
LOG.warn("No remote storage is available, we will default to the first.");
return availableRemoteStorageInfo.values().iterator().next();
}
+
+ // Only for test
+ String getCoordinatorId() {
Review Comment:
Let's move this method into org.apache.uniffle.coordinator.strategy.storage.AbstractSelectStorageStrategy to avoid duplicate code. And the following lines should also be in the `AbstractSelectStorageStrategy`
```
this.coordinatorId = conf.getString(CoordinatorUtils.COORDINATOR_ID,
String.valueOf(CoordinatorUtils.getRandomInt()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] advancedxy merged pull request #648: [#647] fix: Multiple coordinator produce conflicts when they delect the same file
Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy merged PR #648:
URL: https://github.com/apache/incubator-uniffle/pull/648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org