You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2024/04/10 08:12:00 UTC
[PR] [#1634] fix(server): remove app folder if app is expired [incubator-uniffle]
zuston opened a new pull request, #1635:
URL: https://github.com/apache/incubator-uniffle/pull/1635
### What changes were proposed in this pull request?
Always executing the removeResources no matter whether blockIds remain.
### Why are the changes needed?
Fix: #1634
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Using existing unit tests, but additionally checking the app folder after app is expired.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1634] fix(server): remove app folder if app is expired [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #1635:
URL: https://github.com/apache/incubator-uniffle/pull/1635#issuecomment-2046851828
Could you help review this? @rickyma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1634] fix(server): remove app folder if app is expired [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1635:
URL: https://github.com/apache/incubator-uniffle/pull/1635#issuecomment-2046911851
## Test Results
2 107 files - 256 2 107 suites - 256 2h 27m 24s :stopwatch: - 2h 2m 27s
840 tests - 72 829 :white_check_mark: - 82 1 :zzz: ±0 6 :x: + 6 4 :fire: + 4
10 060 runs - 525 9 919 :white_check_mark: - 652 14 :zzz: ±0 71 :x: +71 56 :fire: +56
For more details on these failures and errors, see [this check](https://github.com/apache/incubator-uniffle/runs/23650270691).
Results for commit a86042f1. ± Comparison against base commit a7a0b4ce.
<details>
<summary>This pull request <b>removes</b> 72 tests.</summary>
```
org.apache.uniffle.test.AQERepartitionTest ‑ resultCompareTest
org.apache.uniffle.test.AQESkewedJoinTest ‑ resultCompareTest
org.apache.uniffle.test.AQESkewedJoinWithLocalOrderTest ‑ resultCompareTest
org.apache.uniffle.test.AutoAccessTest ‑ test
org.apache.uniffle.test.CombineByKeyTest ‑ combineByKeyTest
org.apache.uniffle.test.ContinuousSelectPartitionStrategyTest ‑ resultCompareTest
org.apache.uniffle.test.DynamicConfTest ‑ dynamicConfTest
org.apache.uniffle.test.DynamicFetchClientConfTest ‑ test
org.apache.uniffle.test.FailingTasksTest ‑ testFailedTasks
org.apache.uniffle.test.GetReaderTest ‑ test
…
```
</details>
[test-results]:data:application/gzip;base64,H4sIAJRPFmYC/03Nyw7CIBCF4VdpWLuYIQiML2NaLgmxF0Nh1fjuDkmpLs8H/Bwipjns4jFIBHMbxF5T+d++5rGkbWWxVikWPi7tglXQ13OvzjWS9KNXejPhBXFMM4O+IOS8ZZYWzXVtTQTQcM4eJUK66Iz2Jz1qsEOP3ts/bluWVHiJ0WpQMiJGNMEgeINTnDwZAutClAQUlPUkPl8JAjleEgEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1634] fix(server): remove app folder if app is expired [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on code in PR #1635:
URL: https://github.com/apache/incubator-uniffle/pull/1635#discussion_r1559283093
##########
server/src/main/java/org/apache/uniffle/server/ShuffleTaskManager.java:
##########
@@ -763,22 +762,18 @@ public void removeResources(String appId, boolean checkAppExpired) {
return;
}
final long start = System.currentTimeMillis();
- String user = getUserByAppId(appId);
ShuffleTaskInfo shuffleTaskInfo = shuffleTaskInfos.remove(appId);
if (shuffleTaskInfo == null) {
LOG.info("Resource for appId[" + appId + "] had been removed before.");
return;
}
- final Map<Integer, Roaring64NavigableMap> shuffleToCachedBlockIds =
- shuffleTaskInfo.getCachedBlockIds();
partitionsToBlockIds.remove(appId);
shuffleBufferManager.removeBuffer(appId);
shuffleFlushManager.removeResources(appId);
- if (!shuffleToCachedBlockIds.isEmpty()) {
- storageManager.removeResources(
- new AppPurgeEvent(appId, user, new ArrayList<>(shuffleToCachedBlockIds.keySet())));
- }
+
Review Comment:
Useless newline remove it?
##########
server/src/test/java/org/apache/uniffle/server/ShuffleTaskManagerTest.java:
##########
@@ -470,7 +470,11 @@ public void removeShuffleDataWithHdfsTest() throws Exception {
assertTrue(fs.exists(new Path(appBasePath)));
assertNull(shuffleBufferManager.getBufferPool().get(appId).get(0));
assertNotNull(shuffleBufferManager.getBufferPool().get(appId).get(1));
+
+ // the shufflePurgeEvent only will delete the children folders
+ // Once the app is expired, all the app folder should be deleted.
Review Comment:
folder -> folders
##########
server/src/test/java/org/apache/uniffle/server/ShuffleTaskManagerTest.java:
##########
@@ -528,6 +532,14 @@ public void removeShuffleDataWithLocalfileTest() throws Exception {
assertEquals(0, files.length);
}
}
+
+ // the shufflePurgeEvent only will delete the children folders
+ // Once the app is expired, all the app folder should be deleted.
Review Comment:
folder -> folders
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1634] fix(server): remove app folder if app is expired [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #1635:
URL: https://github.com/apache/incubator-uniffle/pull/1635
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org