You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "tsreaper (via GitHub)" <gi...@apache.org> on 2025/06/10 07:11:56 UTC
[PR] [core] Total buckets should remain the same within the same partition when commit [paimon]
tsreaper opened a new pull request, #5725:
URL: https://github.com/apache/paimon/pull/5725
### Purpose
This PR adds a check before commit, that total buckets should remain the same within the same partition.
This check is to avoid the following problem: If the user runs two compact actions with different bucket numbers on a postpone bucket table, there will be files with different total buckets in the same partition, and the same primary key may appear in two buckets.
### Tests
* `TableCommitTest#testGiveUpCommitWhenTotalBucketsChanged`
### API and Format
No format changes.
### Documentation
No new feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Total buckets should remain the same within the same partition when commit [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5725:
URL: https://github.com/apache/paimon/pull/5725#discussion_r2137496635
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1252,10 +1257,47 @@ private List<SimpleFileEntry> readAllEntriesFromChangedPartitions(
private void noConflictsOrFail(
String baseCommitUser,
List<SimpleFileEntry> baseEntries,
- List<SimpleFileEntry> changes) {
+ List<SimpleFileEntry> changes,
+ Snapshot.CommitKind commitKind) {
List<SimpleFileEntry> allEntries = new ArrayList<>(baseEntries);
allEntries.addAll(changes);
+ if (commitKind != Snapshot.CommitKind.OVERWRITE) {
Review Comment:
Why we need to exclude overwrite? Overwrite has no old files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Total buckets should remain the same within the same partition when commit [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #5725:
URL: https://github.com/apache/paimon/pull/5725#discussion_r2137496635
##########
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java:
##########
@@ -1252,10 +1257,47 @@ private List<SimpleFileEntry> readAllEntriesFromChangedPartitions(
private void noConflictsOrFail(
String baseCommitUser,
List<SimpleFileEntry> baseEntries,
- List<SimpleFileEntry> changes) {
+ List<SimpleFileEntry> changes,
+ Snapshot.CommitKind commitKind) {
List<SimpleFileEntry> allEntries = new ArrayList<>(baseEntries);
allEntries.addAll(changes);
+ if (commitKind != Snapshot.CommitKind.OVERWRITE) {
Review Comment:
Why we need to exclude overwrite? Overwrite has no old files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Total buckets should remain the same within the same partition when commit [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #5725:
URL: https://github.com/apache/paimon/pull/5725
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org