You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "laskoviymishka (via GitHub)" <gi...@apache.org> on 2026/06/24 13:19:13 UTC
[PR] fix(table): detect concurrently-removed data files in row-level deletes [iceberg-go]
laskoviymishka opened a new pull request, #1298:
URL: https://github.com/apache/iceberg-go/pull/1298
Two related fixes for optimistic-concurrency validation of position deletes.
1. validateDataFilesExist counted a referenced data file as present whenever any manifest entry for its path existed on the branch head, because it iterated head.dataFiles (discardDeleted=false). A concurrent compaction or overwrite that rewrote the file leaves a DELETED tombstone, so the check passed and the orphaned position delete (its rows had moved to a new file) was committed silently. Iterate data-manifest entries and treat a DELETED entry as absent — only a live ADDED/EXISTING entry satisfies existence, mirroring Java's MergingSnapshotProducer. This corrects the check for every caller, including RowDelta.
2. performMergeOnReadDeletion never ran that check at all: it builds an overwriteFiles producer whose only validator checks partition-filter overlap, and appended its position deletes without registering validateDataFilesExist (wired only into RowDelta.validate). So a merge-on-read delete whose referenced data file was concurrently removed went undetected. writePositionDeletesForFiles now returns the referenced data-file paths of the position deletes it appends, and performMergeOnReadDeletion registers a validateDataFilesExist validator over them, mirroring RowDelta.validate: unconditional (no isolation gating) and additive to the producer's partition-filter validator. A position delete that does not record its referenced data file is skipped, matching RowDelta and Java.
Tests: TestValidateDataFilesExist_TreatsConcurrentlyRemovedFileAsMissing pins the status-aware check (a compacted-away file is missing; a live file is present); TestMergeOnReadDeleteConflict_ReferencedDataFileRemoved drives a merge-on-read delete whose referenced file a concurrent compaction removes and asserts the commit fails with ErrDataFilesMissing; the no-conflict control still commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): detect concurrently-removed data files in row-level deletes [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #1298:
URL: https://github.com/apache/iceberg-go/pull/1298#discussion_r3482956760
##########
table/mor_delete_conflict_test.go:
##########
@@ -0,0 +1,130 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package table_test
+
+import (
+ "context"
+ "path/filepath"
+ "testing"
+
+ "github.com/apache/arrow-go/v18/arrow"
+ "github.com/apache/arrow-go/v18/arrow/array"
+ "github.com/apache/arrow-go/v18/arrow/memory"
+ "github.com/apache/iceberg-go"
+ iceio "github.com/apache/iceberg-go/io"
+ "github.com/apache/iceberg-go/table"
+ "github.com/stretchr/testify/require"
+)
+
+// newV3MoRConflictTestTable builds a v3 table that deletes via merge-on-read
+// (so position deletes are written as Puffin deletion vectors that record their
+// referenced data file) and never retries commits, so a single conflict surfaces
Review Comment:
nit: the doc comment says the table 'never retries commits', but `CommitNumRetriesKey` is set to `2` (with short backoff) just below - the conflict actually surfaces through a retry/refresh rather than by skipping retries. Worth rewording so the comment matches the config (e.g. note it uses short retries so the stale first attempt refreshes and the conflict surfaces terminally).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): detect concurrently-removed data files in row-level deletes [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #1298:
URL: https://github.com/apache/iceberg-go/pull/1298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org