You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "raunaqmorarka (via GitHub)" <gi...@apache.org> on 2026/08/21 14:54:37 UTC
[PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
raunaqmorarka opened a new pull request, #17754:
URL: https://github.com/apache/iceberg/pull/17754
Concurrent RowDelta commits that add DVs for the same data file always fail validation with `Found concurrently added DV`, even though position deletes are commutative. Under concurrent DELETE workloads on V3 tables this makes commits fail without any real conflict.
When the RowDelta does not add or remove data files and `validateNoConflictingDeleteFiles` is not enabled, this change merges the concurrently added DV into the operation's DV for the same data file instead of failing. The V3 spec requires a DV to include all deleted positions from the DV it replaces, so the newest concurrent DV for a data file contains the content of all older ones. That DV is removed from the table and its content is merged into the new DV. Pending removals of delete files for that data file are dropped because the concurrent commits already removed them. Merged puffin files written by invalidated commit attempts are deleted on retry and on commit failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "sezruby (via GitHub)" <gi...@apache.org>.
sezruby commented on PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#issuecomment-5879103577
Gentle nudge on this one 🙏 @nastra @amogh-jahagirdar — as the primary maintainers of the commit-path code here (`MergingSnapshotProducer` / `RowDelta`), your eyes would be ideal. @huaxingao has reviewed end-to-end and it looks correct; the one open review comment (FileIO-resolved cleanup location) is addressed and test coverage is thorough. Whenever you have a spare cycle, a review pass would be much appreciated — no rush, and thanks for all your work on Iceberg!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#issuecomment-5737652881
@amogh-jahagirdar @nastra could you please take a look? This relaxes the "Found concurrently added DV" failure for pure RowDelta deletes and merges the concurrent DV instead. I've reviewed it and it looks correct to me, but a second pair of eyes on the commit-path semantics would be good. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#discussion_r3839625285
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -1183,12 +1289,33 @@ private List<DeleteFile> mergeDVs() {
String.format(
"merged-dvs-%s-%s", snapshotId(), dvMergeAttempt.incrementAndGet())));
- return DVUtil.mergeAndWriteDVsIfRequired(
- dvsByReferencedFile,
- dvOutputLocation,
- fileIO,
- ops().current().specsById(),
- ThreadPools.getDeleteWorkerPool());
+ List<DeleteFile> mergedDVs =
+ DVUtil.mergeAndWriteDVsIfRequired(
+ dvsByReferencedFile,
+ dvOutputLocation,
+ fileIO,
+ ops().current().specsById(),
+ ThreadPools.getDeleteWorkerPool());
+
+ if (requiresMerge) {
+ mergedDVFileLocations.add(dvOutputLocation);
Review Comment:
The merged-DV cleanup tracks the raw location string, mergedDVFileLocations.add(dvOutputLocation) (from locationProvider().newDataLocation(...)), and later deletes it via deleteFile(dvOutputLocation). The overlapping #17309 deliberately tracks the FileIO-resolved fileIO.newOutputFile(dvOutputLocation).location() instead (its comment: the raw location "may differ from … DeleteFile.location() (e.g. stripping a file: URI scheme)"). Deletion by the raw write-string is symmetric for the standard FileIO implementations (local/Hadoop/S3 — testConcurrentDVMergeCleanupOnCommitFailure passes), so this is not a demonstrated leak; but resolving through the FileIO guarantees cleanup targets the canonical written location under any FileIO or a user-supplied deleteWith callback, and aligns with #17309. Recommend adopting the resolved form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#discussion_r4052130531
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -908,19 +955,108 @@ private void validateAddedDVs(
for (ManifestEntry<DeleteFile> entry : entries) {
DeleteFile file = entry.file();
- if (newSnapshotIds.contains(entry.snapshotId()) && ContentFileUtil.isDV(file)) {
+ if (newSnapshotIds.contains(entry.snapshotId())
+ && ContentFileUtil.isDV(file)
+ && dvsByReferencedFile.containsKey(file.referencedDataFile())) {
ValidationException.check(
- !dvsByReferencedFile.containsKey(file.referencedDataFile()),
+ concurrentDVs != null && mergeableSnapshotIds.contains(entry.snapshotId()),
"Found concurrently added DV for %s: %s",
file.referencedDataFile(),
ContentFileUtil.dvDesc(file));
+ concurrentDVs.add(file.copy());
}
}
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
+ /**
+ * Merges concurrently added DVs into this operation's DVs.
+ *
+ * <p>The format requires a DV to include all deleted positions from the DV it replaces, so the
+ * newest concurrent DV for a data file contains the content of all older DVs for that file. That
+ * DV is removed from the table and its content is merged into this operation's DV for the same
+ * data file. Pending removals of delete files for that data file are dropped because the
+ * concurrent commits have already removed them.
+ *
+ * <p>A merge from an earlier commit attempt is undone when the concurrent DV it merged is no
+ * longer present, which happens when the commit that added it is replaced or rolled back.
+ */
+ private void mergeConcurrentDVs(List<DeleteFile> concurrentDVs) {
+ Map<String, DeleteFile> newestDVByReferencedFile = Maps.newHashMap();
+ for (DeleteFile dv : concurrentDVs) {
+ newestDVByReferencedFile.merge(
+ dv.referencedDataFile(),
+ dv,
+ (dv1, dv2) -> dv1.dataSequenceNumber() >= dv2.dataSequenceNumber() ? dv1 : dv2);
+ }
+
+ for (String referencedDataFile : ImmutableList.copyOf(mergedConcurrentDVsByFile.keySet())) {
+ MergedConcurrentDV merged = mergedConcurrentDVsByFile.get(referencedDataFile);
+ DeleteFile concurrentDV = newestDVByReferencedFile.get(referencedDataFile);
+ if (concurrentDV == null || !isSameDV(merged.concurrentDV(), concurrentDV)) {
+ undoConcurrentDVMerge(referencedDataFile, merged);
+ }
+ }
+
+ for (Map.Entry<String, DeleteFile> entry : newestDVByReferencedFile.entrySet()) {
+ String referencedDataFile = entry.getKey();
+ DeleteFile concurrentDV = entry.getValue();
+
+ if (mergedConcurrentDVsByFile.containsKey(referencedDataFile)) {
+ // already merged in a previous commit attempt
+ continue;
+ }
+
+ LOG.info(
+ "Merging concurrently added DV {} for {} into the new DV in table {}",
+ concurrentDV.location(),
+ referencedDataFile,
+ tableName);
+
+ List<DeleteFile> droppedRemovals =
+ deleteFilterManager.filesToBeDeleted().stream()
+ .filter(file -> referencedDataFile.equals(file.referencedDataFile()))
+ .collect(ImmutableList.toImmutableList());
+ droppedRemovals.forEach(deleteFilterManager::dropDelete);
+
+ DeleteFile pendingDV = Delegates.pendingDeleteFile(concurrentDV, null);
Review Comment:
Added a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "sezruby (via GitHub)" <gi...@apache.org>.
sezruby commented on PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#issuecomment-5671877270
cc @huaxingao — you pointed to this PR on the dev@ thread on row-level concurrency for deletion vectors. I've reviewed it end to end and it looks ready to land: the one open review comment (tracking the FileIO-resolved cleanup location) is addressed, and the test coverage is thorough. The position-granular extension to UPDATE/MERGE I'm working on builds directly on this, so it'd be great if you could do a review pass to help it move forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#discussion_r3840023685
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -1183,12 +1289,33 @@ private List<DeleteFile> mergeDVs() {
String.format(
"merged-dvs-%s-%s", snapshotId(), dvMergeAttempt.incrementAndGet())));
- return DVUtil.mergeAndWriteDVsIfRequired(
- dvsByReferencedFile,
- dvOutputLocation,
- fileIO,
- ops().current().specsById(),
- ThreadPools.getDeleteWorkerPool());
+ List<DeleteFile> mergedDVs =
+ DVUtil.mergeAndWriteDVsIfRequired(
+ dvsByReferencedFile,
+ dvOutputLocation,
+ fileIO,
+ ops().current().specsById(),
+ ThreadPools.getDeleteWorkerPool());
+
+ if (requiresMerge) {
+ mergedDVFileLocations.add(dvOutputLocation);
Review Comment:
Changed to track the FileIO-resolved location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "raunaqmorarka (via GitHub)" <gi...@apache.org>.
raunaqmorarka commented on code in PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#discussion_r4035071761
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -908,19 +949,80 @@ private void validateAddedDVs(
for (ManifestEntry<DeleteFile> entry : entries) {
DeleteFile file = entry.file();
- if (newSnapshotIds.contains(entry.snapshotId()) && ContentFileUtil.isDV(file)) {
+ if (newSnapshotIds.contains(entry.snapshotId())
+ && ContentFileUtil.isDV(file)
+ && dvsByReferencedFile.containsKey(file.referencedDataFile())) {
ValidationException.check(
- !dvsByReferencedFile.containsKey(file.referencedDataFile()),
+ concurrentDVs != null && mergeableSnapshotIds.contains(entry.snapshotId()),
"Found concurrently added DV for %s: %s",
file.referencedDataFile(),
ContentFileUtil.dvDesc(file));
+ concurrentDVs.add(file.copy());
}
}
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
+ /**
+ * Merges concurrently added DVs into this operation's DVs.
+ *
+ * <p>The format requires a DV to include all deleted positions from the DV it replaces, so the
+ * newest concurrent DV for a data file contains the content of all older DVs for that file. That
+ * DV is removed from the table and its content is merged into this operation's DV for the same
+ * data file. Pending removals of delete files for that data file are dropped because the
+ * concurrent commits have already removed them.
+ */
+ private void mergeConcurrentDVs(List<DeleteFile> concurrentDVs) {
+ Map<String, DeleteFile> newestDVByReferencedFile = Maps.newHashMap();
+ for (DeleteFile dv : concurrentDVs) {
+ newestDVByReferencedFile.merge(
+ dv.referencedDataFile(),
+ dv,
+ (dv1, dv2) -> dv1.dataSequenceNumber() >= dv2.dataSequenceNumber() ? dv1 : dv2);
+ }
+
+ for (Map.Entry<String, DeleteFile> entry : newestDVByReferencedFile.entrySet()) {
+ String referencedDataFile = entry.getKey();
+ DeleteFile concurrentDV = entry.getValue();
+
+ DeleteFile previouslyMerged = mergedConcurrentDVsByFile.get(referencedDataFile);
+ if (previouslyMerged != null) {
+ if (isSameDV(previouslyMerged, concurrentDV)) {
+ // already merged in a previous commit attempt
+ continue;
+ }
+
+ // the previously merged DV was replaced by a newer commit that merged its content
+ dvsByReferencedFile.get(referencedDataFile).remove(previouslyMerged);
Review Comment:
Confirmed with a test, the rollback left the first attempt's merge in place. A merge from an earlier attempt is now undone when its concurrent DV is no longer present, which also restores the delete file removals the merge had dropped. Added coverage for a rollback between attempts and for the branch being removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "sezruby (via GitHub)" <gi...@apache.org>.
sezruby commented on PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#issuecomment-6151170114
@rdblue @RussellSpitzer @szehon-ho could one of you take a look at this when you have a chance? It merges a concurrently added DV into the operation's DV for pure RowDelta deletes, instead of failing with `Found concurrently added DV`. You reviewed the related duplicate-DV merge in #15006, so the commit-path context should be familiar.
@huaxingao has reviewed it end to end and asked for a second committer's view on the commit/retry semantics. All open comments are addressed, including the rollback case where the concurrent DV disappears between attempts. It is also the first step of the row-granular DV concurrency proposal on dev@ (#18020). Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "sezruby (via GitHub)" <gi...@apache.org>.
sezruby commented on PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#issuecomment-5987965614
@nastra @amogh-jahagirdar — gentle follow-up on the request for committer feedback. @uros-b noted in the September 29 review that the change makes sense but should wait for additional feedback from committers: https://github.com/apache/iceberg/pull/17754#pullrequestreview-5358050589
Could either of you take a look at the concurrent-DV merge for pure RowDelta deletes, particularly the commit/retry semantics, when you have a chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#discussion_r4034049102
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -908,19 +949,80 @@ private void validateAddedDVs(
for (ManifestEntry<DeleteFile> entry : entries) {
DeleteFile file = entry.file();
- if (newSnapshotIds.contains(entry.snapshotId()) && ContentFileUtil.isDV(file)) {
+ if (newSnapshotIds.contains(entry.snapshotId())
+ && ContentFileUtil.isDV(file)
+ && dvsByReferencedFile.containsKey(file.referencedDataFile())) {
ValidationException.check(
- !dvsByReferencedFile.containsKey(file.referencedDataFile()),
+ concurrentDVs != null && mergeableSnapshotIds.contains(entry.snapshotId()),
"Found concurrently added DV for %s: %s",
file.referencedDataFile(),
ContentFileUtil.dvDesc(file));
+ concurrentDVs.add(file.copy());
}
}
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
+ /**
+ * Merges concurrently added DVs into this operation's DVs.
+ *
+ * <p>The format requires a DV to include all deleted positions from the DV it replaces, so the
+ * newest concurrent DV for a data file contains the content of all older DVs for that file. That
+ * DV is removed from the table and its content is merged into this operation's DV for the same
+ * data file. Pending removals of delete files for that data file are dropped because the
+ * concurrent commits have already removed them.
+ */
+ private void mergeConcurrentDVs(List<DeleteFile> concurrentDVs) {
+ Map<String, DeleteFile> newestDVByReferencedFile = Maps.newHashMap();
+ for (DeleteFile dv : concurrentDVs) {
+ newestDVByReferencedFile.merge(
+ dv.referencedDataFile(),
+ dv,
+ (dv1, dv2) -> dv1.dataSequenceNumber() >= dv2.dataSequenceNumber() ? dv1 : dv2);
+ }
+
+ for (Map.Entry<String, DeleteFile> entry : newestDVByReferencedFile.entrySet()) {
+ String referencedDataFile = entry.getKey();
+ DeleteFile concurrentDV = entry.getValue();
+
+ DeleteFile previouslyMerged = mergedConcurrentDVsByFile.get(referencedDataFile);
+ if (previouslyMerged != null) {
+ if (isSameDV(previouslyMerged, concurrentDV)) {
+ // already merged in a previous commit attempt
+ continue;
+ }
+
+ // the previously merged DV was replaced by a newer commit that merged its content
+ dvsByReferencedFile.get(referencedDataFile).remove(previouslyMerged);
Review Comment:
I think this handles the concurrent DV being replaced between attempts, but not it disappearing.
For example, a data file has rows 0-3. Alice deletes 0,1 and commits a DV. Bob deletes 2,3. Bob's attempt 1 merges Alice's DV, so his DV now covers 0,1,2,3, then his commit fails and retries. In between, the branch is rolled back to before Alice's commit.
Attempt 2 finds no concurrent DV, so line 933 skips mergeConcurrentDVs and attempt 1's merge is never reconsidered — dvsByReferencedFile is never cleared, and the retry calls apply() on the same instance. Bob still points at Alice's DV file, which the rollback didn't delete, so he commits 0,1,2,3. Rows 0 and 1 end up deleted even though Alice's delete was rolled back and Bob never asked for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "sezruby (via GitHub)" <gi...@apache.org>.
sezruby commented on PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#issuecomment-5637541826
Reviewed this end to end. The concurrent-DV merge is correct — the union preserves all deleted positions given the one-DV-per-file invariant, and restricting merges to DELETE-sourced snapshots (`mergeableSnapshotIds`) correctly excludes overwrite/replace-added DVs. Test coverage is comprehensive: retry idempotency, a concurrent DV replaced between attempts (with the prior merged file's deletion asserted), cleanup on commit failure, and the fail-closed cases for adds/removes-data-files and overwrite-sourced DVs.
@uros-b the FileIO-resolved cleanup-location comment looks addressed — would you mind another look? This is a clean base to extend row-level concurrency to UPDATE/MERGE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#discussion_r4051537521
##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -908,19 +955,108 @@ private void validateAddedDVs(
for (ManifestEntry<DeleteFile> entry : entries) {
DeleteFile file = entry.file();
- if (newSnapshotIds.contains(entry.snapshotId()) && ContentFileUtil.isDV(file)) {
+ if (newSnapshotIds.contains(entry.snapshotId())
+ && ContentFileUtil.isDV(file)
+ && dvsByReferencedFile.containsKey(file.referencedDataFile())) {
ValidationException.check(
- !dvsByReferencedFile.containsKey(file.referencedDataFile()),
+ concurrentDVs != null && mergeableSnapshotIds.contains(entry.snapshotId()),
"Found concurrently added DV for %s: %s",
file.referencedDataFile(),
ContentFileUtil.dvDesc(file));
+ concurrentDVs.add(file.copy());
}
}
} catch (IOException e) {
throw new UncheckedIOException(e);
}
}
+ /**
+ * Merges concurrently added DVs into this operation's DVs.
+ *
+ * <p>The format requires a DV to include all deleted positions from the DV it replaces, so the
+ * newest concurrent DV for a data file contains the content of all older DVs for that file. That
+ * DV is removed from the table and its content is merged into this operation's DV for the same
+ * data file. Pending removals of delete files for that data file are dropped because the
+ * concurrent commits have already removed them.
+ *
+ * <p>A merge from an earlier commit attempt is undone when the concurrent DV it merged is no
+ * longer present, which happens when the commit that added it is replaced or rolled back.
+ */
+ private void mergeConcurrentDVs(List<DeleteFile> concurrentDVs) {
+ Map<String, DeleteFile> newestDVByReferencedFile = Maps.newHashMap();
+ for (DeleteFile dv : concurrentDVs) {
+ newestDVByReferencedFile.merge(
+ dv.referencedDataFile(),
+ dv,
+ (dv1, dv2) -> dv1.dataSequenceNumber() >= dv2.dataSequenceNumber() ? dv1 : dv2);
+ }
+
+ for (String referencedDataFile : ImmutableList.copyOf(mergedConcurrentDVsByFile.keySet())) {
+ MergedConcurrentDV merged = mergedConcurrentDVsByFile.get(referencedDataFile);
+ DeleteFile concurrentDV = newestDVByReferencedFile.get(referencedDataFile);
+ if (concurrentDV == null || !isSameDV(merged.concurrentDV(), concurrentDV)) {
+ undoConcurrentDVMerge(referencedDataFile, merged);
+ }
+ }
+
+ for (Map.Entry<String, DeleteFile> entry : newestDVByReferencedFile.entrySet()) {
+ String referencedDataFile = entry.getKey();
+ DeleteFile concurrentDV = entry.getValue();
+
+ if (mergedConcurrentDVsByFile.containsKey(referencedDataFile)) {
+ // already merged in a previous commit attempt
+ continue;
+ }
+
+ LOG.info(
+ "Merging concurrently added DV {} for {} into the new DV in table {}",
+ concurrentDV.location(),
+ referencedDataFile,
+ tableName);
+
+ List<DeleteFile> droppedRemovals =
+ deleteFilterManager.filesToBeDeleted().stream()
+ .filter(file -> referencedDataFile.equals(file.referencedDataFile()))
+ .collect(ImmutableList.toImmutableList());
+ droppedRemovals.forEach(deleteFilterManager::dropDelete);
+
+ DeleteFile pendingDV = Delegates.pendingDeleteFile(concurrentDV, null);
Review Comment:
nit: Maybe add a comment on why null instead of the concurrent DV's own sequence number. `DVUtil.validateCanMerge` requires all DVs for a data file to share a `dataSequenceNumber`, and this operation's DVs are pending (null), so null is what makes the merge legal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Merge concurrently added DVs for RowDelta pure delete commits [iceberg]
Posted by "raunaqmorarka (via GitHub) via issues" <is...@iceberg.apache.org>.
raunaqmorarka commented on PR #17754:
URL: https://github.com/apache/iceberg/pull/17754#issuecomment-6546885049
@amogh-jahagirdar @nastra @RussellSpitzer @aokolnychyi PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org