You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "leaves12138 (via GitHub)" <gi...@apache.org> on 2026/06/13 15:10:11 UTC
[PR] [core] Reuse manifest merge results on commit retry [paimon]
leaves12138 opened a new pull request, #8229:
URL: https://github.com/apache/paimon/pull/8229
## Summary
- Reuse the previous manifest merge result on commit retry when all previously merged input manifests still exist in the latest snapshot.
- Replace those input manifests in the latest manifest list with the merged output manifests, preserving concurrent manifests and falling back to a fresh merge if any input manifest disappeared.
- Always write a fresh base manifest list for the retry commit so no base manifest list file is reused across attempts.
- Add retry conflict tests for contiguous, non-contiguous, missing-before, multi-retry, empty-table, and delete-order cases.
## Tests
- `mvn -pl paimon-core -Pfast-build -Dtest=FileStoreCommitTest#testCommitRetryReusePreviousManifestMergeResultWhenBeforeStillExists+testCommitRetryReusePreviousManifestMergeResultWhenBeforeExistsNonContiguously+testCommitRetryDoesNotReusePreviousManifestMergeResultWhenBeforeWasMerged+testCommitRetryReusePreviousManifestMergeResultAcrossMultipleRetries+testCommitRetryFromEmptyTableWithConcurrentFirstSnapshot+testCommitRetryReusePreviousManifestMergeResultPreservesDeleteOrder test`
- `mvn -pl paimon-core -Pfast-build -Dtest=FileStoreCommitTest test`
- `mvn -pl paimon-core -DskipTests compile`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reuse manifest merge results on commit retry [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8229:
URL: https://github.com/apache/paimon/pull/8229#issuecomment-4756559354
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reuse manifest merge results on commit retry [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #8229:
URL: https://github.com/apache/paimon/pull/8229#discussion_r3433997461
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1123,6 +1141,81 @@ CommitResult tryCommitOnce(
return new SuccessCommitResult();
}
+ @Nullable
+ private ManifestMergeReuse tryReuseManifestMergeResult(
+ @Nullable RetryCommitResult retryResult, List<ManifestFileMeta> currentManifests) {
+ if (!(retryResult instanceof CommitFailRetryResult)) {
+ return null;
+ }
+
+ CommitFailRetryResult commitFailRetry = (CommitFailRetryResult) retryResult;
+ ManifestMergeResult previous = commitFailRetry.manifestMergeResult;
+ if (previous == null) {
+ return null;
+ }
+ if (previous.mergeBeforeManifests.isEmpty() && !currentManifests.isEmpty()) {
+ return null;
+ }
+
+ List<ManifestFileMeta> mergeAfterManifests =
+ replaceMergedManifests(
+ currentManifests,
+ previous.mergeBeforeManifests,
+ previous.mergeAfterManifests);
+ if (mergeAfterManifests == null) {
+ return null;
+ }
+
+ return new ManifestMergeReuse(currentManifests, mergeAfterManifests);
+ }
+
+ @Nullable
+ private List<ManifestFileMeta> replaceMergedManifests(
+ List<ManifestFileMeta> currentManifests,
+ List<ManifestFileMeta> mergeBeforeManifests,
+ List<ManifestFileMeta> mergedManifests) {
+ List<ManifestFileMeta> remainingMergeBefore = new ArrayList<>(mergeBeforeManifests);
+ List<ManifestFileMeta> replacementManifests =
+ new ArrayList<>(
+ Math.max(
+ 0,
+ currentManifests.size()
+ - mergeBeforeManifests.size()
+ + mergedManifests.size()));
+ boolean insertedMergeAfter = false;
+ for (ManifestFileMeta currentManifest : currentManifests) {
+ int mergeBeforeIndex = remainingMergeBefore.indexOf(currentManifest);
Review Comment:
You can use ListUtils something like `list.replaceAll`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reuse manifest merge results on commit retry [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8229:
URL: https://github.com/apache/paimon/pull/8229#issuecomment-4730193790
Sounds no issue from me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reuse manifest merge results on commit retry [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8229:
URL: https://github.com/apache/paimon/pull/8229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org