You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "xxubai (via GitHub)" <gi...@apache.org> on 2025/06/19 02:26:58 UTC
[PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
xxubai opened a new pull request, #3630:
URL: https://github.com/apache/amoro/pull/3630
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://amoro.apache.org/how-to-contribute/
2. If the PR is related to an issue in https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title, e.g., '[AMORO-XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][AMORO-XXXX] Your PR title ...'.
-->
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related issue
-->
Subtask of #3608
## Brief change log
<!--
Clearly describe the changes made in modules, classes, methods, etc.
-->
- As title
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #3630:
URL: https://github.com/apache/amoro/pull/3630#discussion_r2342876485
##########
amoro-ams/src/main/java/org/apache/amoro/server/utils/RollingFileCleaner.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.server.utils;
+
+import org.apache.amoro.io.AuthenticatedFileIO;
+import org.apache.amoro.shade.guava32.com.google.common.collect.Sets;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.io.BulkDeletionFailureException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RollingFileCleaner {
+ private final Set<String> collectedFiles = Sets.newConcurrentHashSet();
+ private final Set<String> excludeFiles;
+
+ private static final int CLEANED_FILE_GROUP_SIZE = 1_000;
+
+ private final AtomicInteger fileCounter = new AtomicInteger(0);
+ private final AtomicInteger cleanedFileCounter = new AtomicInteger(0);
+
+ private final AuthenticatedFileIO fileIO;
+
+ private static final Logger LOG = LoggerFactory.getLogger(RollingFileCleaner.class);
+
+ public RollingFileCleaner(AuthenticatedFileIO fileIO, Set<String> excludeFiles) {
+ this.fileIO = fileIO;
+ this.excludeFiles =
+ excludeFiles != null
+ ? Sets.newConcurrentHashSet(excludeFiles)
+ : Sets.newConcurrentHashSet();
+ }
+
+ public void addFile(String filePath) {
+ if (excludeFiles.isEmpty()) {
+ collectedFiles.add(filePath);
+ int fc = fileCounter.incrementAndGet();
+
+ if (fc % CLEANED_FILE_GROUP_SIZE == 0) {
Review Comment:
It seems code 58-59 and 70-71 are duplicates; maybe we can move them out of the if-else code block.
##########
amoro-ams/src/main/java/org/apache/amoro/server/utils/RollingFileCleaner.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.server.utils;
+
+import org.apache.amoro.io.AuthenticatedFileIO;
+import org.apache.amoro.shade.guava32.com.google.common.collect.Sets;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.io.BulkDeletionFailureException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RollingFileCleaner {
+ private final Set<String> collectedFiles = Sets.newConcurrentHashSet();
+ private final Set<String> excludeFiles;
+
+ private static final int CLEANED_FILE_GROUP_SIZE = 1_000;
+
+ private final AtomicInteger fileCounter = new AtomicInteger(0);
+ private final AtomicInteger cleanedFileCounter = new AtomicInteger(0);
+
+ private final AuthenticatedFileIO fileIO;
+
+ private static final Logger LOG = LoggerFactory.getLogger(RollingFileCleaner.class);
+
+ public RollingFileCleaner(AuthenticatedFileIO fileIO, Set<String> excludeFiles) {
+ this.fileIO = fileIO;
+ this.excludeFiles =
+ excludeFiles != null
+ ? Sets.newConcurrentHashSet(excludeFiles)
+ : Sets.newConcurrentHashSet();
+ }
+
+ public void addFile(String filePath) {
+ if (excludeFiles.isEmpty()) {
Review Comment:
Maybe we should collect the parent paths and try to delete them at last to avoid keeping themempty directorties here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3630:
URL: https://github.com/apache/amoro/pull/3630#issuecomment-2986517984
## [Codecov](https://app.codecov.io/gh/apache/amoro/pull/3630?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `83.58209%` with `11 lines` in your changes missing coverage. Please review.
> Project coverage is 28.34%. Comparing base [(`7197ec6`)](https://app.codecov.io/gh/apache/amoro/commit/7197ec697b66bca5b8f29fbcae6f2d4898e9e978?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`34772ea`)](https://app.codecov.io/gh/apache/amoro/commit/34772ea5458e85b3a7dcca19f9d87c3d916e7227?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 5 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/amoro/pull/3630?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../apache/amoro/server/utils/RollingFileCleaner.java](https://app.codecov.io/gh/apache/amoro/pull/3630?src=pr&el=tree&filepath=amoro-ams%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Famoro%2Fserver%2Futils%2FRollingFileCleaner.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-YW1vcm8tYW1zL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9hbW9yby9zZXJ2ZXIvdXRpbHMvUm9sbGluZ0ZpbGVDbGVhbmVyLmphdmE=) | 79.62% | [6 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/amoro/pull/3630?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3630 +/- ##
============================================
+ Coverage 21.76% 28.34% +6.57%
- Complexity 2391 3722 +1331
============================================
Files 436 618 +182
Lines 40498 49883 +9385
Branches 5743 6440 +697
============================================
+ Hits 8816 14140 +5324
- Misses 30935 34730 +3795
- Partials 747 1013 +266
```
| [Flag](https://app.codecov.io/gh/apache/amoro/pull/3630/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core](https://app.codecov.io/gh/apache/amoro/pull/3630/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `28.34% <83.58%> (?)` | |
| [trino](https://app.codecov.io/gh/apache/amoro/pull/3630/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/amoro/pull/3630?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3630:
URL: https://github.com/apache/amoro/pull/3630#issuecomment-3256520397
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on PR #3630:
URL: https://github.com/apache/amoro/pull/3630#issuecomment-3291201079
It seems there are some CI issues that need to be fixed. @xxubai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on code in PR #3630:
URL: https://github.com/apache/amoro/pull/3630#discussion_r2348366429
##########
amoro-ams/src/main/java/org/apache/amoro/server/utils/RollingFileCleaner.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.server.utils;
+
+import org.apache.amoro.io.AuthenticatedFileIO;
+import org.apache.amoro.shade.guava32.com.google.common.collect.Sets;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.io.BulkDeletionFailureException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RollingFileCleaner {
+ private final Set<String> collectedFiles = Sets.newConcurrentHashSet();
+ private final Set<String> excludeFiles;
+
+ private static final int CLEANED_FILE_GROUP_SIZE = 1_000;
+
+ private final AtomicInteger fileCounter = new AtomicInteger(0);
+ private final AtomicInteger cleanedFileCounter = new AtomicInteger(0);
+
+ private final AuthenticatedFileIO fileIO;
+
+ private static final Logger LOG = LoggerFactory.getLogger(RollingFileCleaner.class);
+
+ public RollingFileCleaner(AuthenticatedFileIO fileIO, Set<String> excludeFiles) {
+ this.fileIO = fileIO;
+ this.excludeFiles =
+ excludeFiles != null
+ ? Sets.newConcurrentHashSet(excludeFiles)
+ : Sets.newConcurrentHashSet();
+ }
+
+ public void addFile(String filePath) {
+ if (excludeFiles.isEmpty()) {
Review Comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #3630:
URL: https://github.com/apache/amoro/pull/3630#discussion_r2348300082
##########
amoro-ams/src/main/java/org/apache/amoro/server/utils/RollingFileCleaner.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.amoro.server.utils;
+
+import org.apache.amoro.io.AuthenticatedFileIO;
+import org.apache.amoro.shade.guava32.com.google.common.collect.Sets;
+import org.apache.hadoop.fs.Path;
+import org.apache.iceberg.io.BulkDeletionFailureException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.net.URI;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicInteger;
+
+public class RollingFileCleaner {
+ private final Set<String> collectedFiles = Sets.newConcurrentHashSet();
+ private final Set<String> excludeFiles;
+
+ private static final int CLEANED_FILE_GROUP_SIZE = 1_000;
+
+ private final AtomicInteger fileCounter = new AtomicInteger(0);
+ private final AtomicInteger cleanedFileCounter = new AtomicInteger(0);
+
+ private final AuthenticatedFileIO fileIO;
+
+ private static final Logger LOG = LoggerFactory.getLogger(RollingFileCleaner.class);
+
+ public RollingFileCleaner(AuthenticatedFileIO fileIO, Set<String> excludeFiles) {
+ this.fileIO = fileIO;
+ this.excludeFiles =
+ excludeFiles != null
+ ? Sets.newConcurrentHashSet(excludeFiles)
+ : Sets.newConcurrentHashSet();
+ }
+
+ public void addFile(String filePath) {
+ if (excludeFiles.isEmpty()) {
Review Comment:
add remove empty dirs in `doCleanFiles`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3608] Using rolling batch files cleaner to reduce memory usage [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong merged PR #3630:
URL: https://github.com/apache/amoro/pull/3630
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org