You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jerry-024 (via GitHub)" <gi...@apache.org> on 2025/03/11 09:12:49 UTC
[PR] [bug]ignore empty changelog when orphan files [paimon]
jerry-024 opened a new pull request, #5257:
URL: https://github.com/apache/paimon/pull/5257
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close #xxx
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: ignore empty changelog and fix flink job submit slow [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5257:
URL: https://github.com/apache/paimon/pull/5257#discussion_r1994639825
##########
paimon-core/src/main/java/org/apache/paimon/utils/ChangelogManager.java:
##########
@@ -154,7 +154,21 @@ public List<Changelog> safelyGetAllChangelogs() throws IOException {
collectSnapshots(
path -> {
try {
- changelogs.add(Changelog.fromJson(fileIO.readFileUtf8(path)));
+ String changeLog = "";
+ try {
+ changeLog = fileIO.readFileUtf8(path);
+ if (!StringUtils.isNullOrWhitespaceOnly(changeLog)) {
Review Comment:
Maybe we don't need to ignore empty files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: fix flink job submit slow [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #5257:
URL: https://github.com/apache/paimon/pull/5257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: fix flink job submit slow [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #5257:
URL: https://github.com/apache/paimon/pull/5257#issuecomment-2733048291
cc @yunfengzhou-hub to take a look again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: fix flink job submit slow [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #5257:
URL: https://github.com/apache/paimon/pull/5257#issuecomment-2732835122
re-open to trigger test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: fix flink job submit slow [paimon]
Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #5257:
URL: https://github.com/apache/paimon/pull/5257#issuecomment-2735152949
Thanks for the update! LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: ignore empty changelog and fix flink job submit slow [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5257:
URL: https://github.com/apache/paimon/pull/5257#discussion_r1995093679
##########
paimon-core/src/main/java/org/apache/paimon/operation/CleanOrphanFilesResult.java:
##########
@@ -22,15 +22,18 @@
import javax.annotation.Nullable;
+import java.io.Serializable;
import java.util.List;
/** The result of OrphanFilesClean. */
-public class CleanOrphanFilesResult {
+public class CleanOrphanFilesResult implements Serializable {
Review Comment:
Why this class need to be `Serializable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: ignore empty changelog and fix flink job submit slow [paimon]
Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on code in PR #5257:
URL: https://github.com/apache/paimon/pull/5257#discussion_r1995242820
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -93,17 +101,45 @@ public DataStream<CleanOrphanFilesResult> doOrphanClean(StreamExecutionEnvironme
// Flink 1.17 introduced this config, use string to keep compatibility
flinkConf.setString("execution.batch.adaptive.auto-parallelism.enabled", "false");
env.configure(flinkConf);
-
+ LOG.info("Starting orphan files clean for table {}", table.name());
+ long start = System.currentTimeMillis();
List<String> branches = validBranches();
+ LOG.info(
+ "End orphan files validBranches: spend [{}] ms",
+ System.currentTimeMillis() - start);
// snapshot and changelog files are the root of everything, so they are handled specially
// here, and subsequently, we will not count their orphan files.
- AtomicLong deletedFilesCountInLocal = new AtomicLong(0);
- AtomicLong deletedFilesLenInBytesInLocal = new AtomicLong(0);
- cleanSnapshotDir(
- branches,
- path -> deletedFilesCountInLocal.incrementAndGet(),
- deletedFilesLenInBytesInLocal::addAndGet);
+ DataStream<CleanOrphanFilesResult> branchSnapshotDirDeleted =
+ env.fromCollection(branches)
+ .process(
+ new ProcessFunction<String, Tuple2<Long, Long>>() {
+ @Override
+ public void processElement(
+ String branch,
+ ProcessFunction<String, Tuple2<Long, Long>>.Context ctx,
+ Collector<Tuple2<Long, Long>> out)
+ throws Exception {
+ AtomicLong deletedFilesCount = new AtomicLong(0);
+ AtomicLong deletedFilesLenInBytes = new AtomicLong(0);
+ cleanBranchSnapshotDir(
+ branch,
+ path -> deletedFilesCount.incrementAndGet(),
+ deletedFilesLenInBytes::addAndGet);
+ out.collect(
+ new Tuple2<>(
+ deletedFilesCount.get(),
+ deletedFilesLenInBytes.get()));
+ }
+ })
+ .keyBy(tuple -> 1)
+ .reduce(
Review Comment:
it might be better to force set the parallelism of this operator to 1.
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/orphan/FlinkOrphanFilesClean.java:
##########
@@ -202,31 +238,86 @@ public void endInput() throws IOException {
});
usedFiles = usedFiles.union(usedManifestFiles);
-
- List<String> fileDirs =
- listPaimonFileDirs().stream()
- .map(Path::toUri)
- .map(Object::toString)
- .collect(Collectors.toList());
- DataStream<Pair<String, Long>> candidates =
- env.fromCollection(fileDirs)
+ FileStorePathFactory pathFactory = table.store().pathFactory();
+ List<Tuple7<String, String, String, String, String, Integer, String>> tablePaths =
+ Arrays.asList(
+ new Tuple7<>(
+ table.fullName(),
+ pathFactory.manifestPath().toString(),
+ pathFactory.indexPath().toString(),
+ pathFactory.statisticsPath().toString(),
+ pathFactory.dataFilePath().toString(),
+ partitionKeysNum,
+ table.store().options().dataFileExternalPaths()));
+ DataStream<Tuple2<String, Long>> candidates =
+ env.fromCollection(
+ tablePaths,
+ TypeInformation.of(
+ new TypeHint<
+ Tuple7<
+ String,
+ String,
+ String,
+ String,
+ String,
+ Integer,
+ String>>() {}))
Review Comment:
Same as above. since tablePaths.size() is 1, a bigger parallelism would be a waste of resource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: fix flink job submit slow [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #5257: [bug]orphan files: fix flink job submit slow
URL: https://github.com/apache/paimon/pull/5257
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [bug]orphan files: fix flink job submit slow [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #5257:
URL: https://github.com/apache/paimon/pull/5257#issuecomment-2735620063
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org