You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Revanth14 (via GitHub)" <gi...@apache.org> on 2026/10/02 01:38:47 UTC
[PR] fix(table): reject copy-on-write commits that would drop concurrent deletes [iceberg-go]
Revanth14 opened a new pull request, #2099:
URL: https://github.com/apache/iceberg-go/pull/2099
Fixes #2090.
Copy-on-write `Delete` and `Overwrite` could resurrect deleted rows. If a concurrent writer added deletes against a data file the operation removes, and commit retries were enabled, refresh-and-replay committed the stale rewrite without checking those deletes.
`performCopyOnWriteDeletion` now registers `validateNoNewDeletesForRewrittenFiles` over every data file it removes, both rewritten and fully dropped, at every isolation level. This matches Java's `validateNoConflictingDeletes` for copy-on-write. Conflicting commits now fail with `ErrConflictingDeleteFiles`. Because `Delete` (copy-on-write mode) and `Overwrite` share this path, both are covered.
### Conservative rejection (existing validator behavior, unchanged here)
- A position delete that doesn't identify a single target file falls back to a `(spec, partition)` match. On v2, a concurrent position delete against a *different* file in the same partition can therefore reject the commit. Java's `DeleteFileIndex` falls back the same way.
- Any concurrent equality delete is rejected, whatever partition it touches. This is stricter than Java; the validator's doc comment already notes the planned follow-up to narrow it.
### Tests
`table/cow_delete_conflict_test.go`:
- 16 conflict cases: format v2/v3 × serializable/snapshot isolation × `Delete`/`Overwrite` × rewritten/fully removed file. A concurrent merge-on-read delete commits first; the copy-on-write commit must fail with `ErrConflictingDeleteFiles`.
- 8 controls: the same matrix with a concurrent append instead; the copy-on-write commit must replay and succeed with both changes applied.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): reject copy-on-write commits that would drop concurrent deletes [iceberg-go]
Posted by "Revanth14 (via GitHub)" <gi...@apache.org>.
Revanth14 commented on code in PR #2099:
URL: https://github.com/apache/iceberg-go/pull/2099#discussion_r4172180524
##########
table/cow_delete_conflict_test.go:
##########
@@ -0,0 +1,172 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package table_test
+
+import (
+ "context"
+ "fmt"
+ "path/filepath"
+ "testing"
+
+ "github.com/apache/arrow-go/v18/arrow"
+ "github.com/apache/arrow-go/v18/arrow/array"
+ "github.com/apache/arrow-go/v18/arrow/memory"
+ "github.com/apache/iceberg-go"
+ iceio "github.com/apache/iceberg-go/io"
+ "github.com/apache/iceberg-go/table"
+ "github.com/stretchr/testify/require"
+)
+
+// newCoWConflictTestTable builds a table that deletes via merge-on-read (so a
+// concurrent delete writes delete files rather than rewriting data) and
+// retries commits, so a lost CAS race is refreshed and replayed. Both the
+// delete and update isolation levels are set to isolation.
+func newCoWConflictTestTable(t *testing.T, formatVersion string, isolation table.IsolationLevel) *table.Table {
+ t.Helper()
+
+ location := filepath.ToSlash(t.TempDir())
+ schema := iceberg.NewSchema(0,
+ iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+ iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+ )
+ meta, err := table.NewMetadata(schema, iceberg.UnpartitionedSpec, table.UnsortedSortOrder, location,
+ iceberg.Properties{
+ table.PropertyFormatVersion: formatVersion,
+ table.WriteDeleteModeKey: table.WriteModeMergeOnRead,
+ table.WriteDeleteIsolationLevelKey: string(isolation),
+ table.WriteUpdateIsolationLevelKey: string(isolation),
+ table.CommitNumRetriesKey: "2",
+ table.CommitMinRetryWaitMsKey: "1",
+ table.CommitMaxRetryWaitMsKey: "2",
+ table.CommitTotalRetryTimeoutMsKey: "1000",
+ })
+ require.NoError(t, err)
+
+ metaLoc := location + "/metadata/v1.metadata.json"
+ fsF := func(context.Context) (iceio.IO, error) { return iceio.LocalFS{}, nil }
+ cat := &concurrentTestCatalog{metadata: meta, location: metaLoc, fsF: fsF}
+
+ return table.New(table.Identifier{"db", "cow_delete_conflict"}, meta, metaLoc, fsF, cat)
+}
+
+func cowTestRecords(t *testing.T, rowsJSON string) array.RecordReader {
+ t.Helper()
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {Name: "id", Type: arrow.PrimitiveTypes.Int64, Nullable: false},
+ {Name: "data", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+ data, err := array.TableFromJSON(memory.DefaultAllocator, arrowSchema, []string{rowsJSON})
+ require.NoError(t, err)
+ t.Cleanup(data.Release)
+
+ return array.NewTableReader(data, -1)
+}
+
+// stageCopyOnWrite stages, without committing, a copy-on-write removal of the
+// rows matching filter through either Delete or Overwrite (with no new rows).
+func stageCopyOnWrite(t *testing.T, tbl *table.Table, op string, filter iceberg.BooleanExpression) *table.Transaction {
+ t.Helper()
+ ctx := context.Background()
+
+ txn := tbl.NewTransaction()
+ switch op {
+ case "delete":
+ require.NoError(t, txn.SetProperties(iceberg.Properties{
+ table.WriteDeleteModeKey: table.WriteModeCopyOnWrite,
+ }))
+ require.NoError(t, txn.Delete(ctx, filter, nil))
+ case "overwrite":
+ require.NoError(t, txn.Overwrite(ctx, cowTestRecords(t, `[]`), nil, table.WithOverwriteFilter(filter)))
+ default:
+ t.Fatalf("unknown op %q", op)
+ }
+
+ return txn
+}
+
+var (
+ cowFormatVersions = []string{"2", "3"}
+ cowIsolations = []table.IsolationLevel{table.IsolationSerializable, table.IsolationSnapshot}
+ cowOps = []string{"delete", "overwrite"}
+)
+
+// TestCopyOnWriteConflict_ConcurrentDeleteOnRemovedFile proves a copy-on-write
+// Delete or Overwrite is rejected, at every isolation level, when a concurrent
+// commit added a delete against a data file it removes. Without the validator,
+// refresh-and-replay swaps the original file for a rewrite built from the stale
+// snapshot, dropping the concurrent delete and resurrecting its row (#2090).
+func TestCopyOnWriteConflict_ConcurrentDeleteOnRemovedFile(t *testing.T) {
+ removals := []struct {
+ name string
+ filter iceberg.BooleanExpression
+ }{
+ // id==2 matches part of the file, so it is rewritten.
+ {"rewritten", iceberg.EqualTo(iceberg.Reference("id"), int64(2))},
+ // id>=1 matches every row, so the file is dropped outright.
+ {"fully deleted", iceberg.GreaterThanEqual(iceberg.Reference("id"), int64(1))},
+ }
+
+ for _, version := range cowFormatVersions {
+ for _, isolation := range cowIsolations {
+ for _, op := range cowOps {
+ for _, removal := range removals {
+ name := fmt.Sprintf("v%s/%s/%s/%s", version, isolation, op, removal.name)
+ t.Run(name, func(t *testing.T) {
+ ctx := context.Background()
+ tbl := appendTenRows(t, newCoWConflictTestTable(t, version, isolation))
+
+ txn := stageCopyOnWrite(t, tbl, op, removal.filter)
+
+ // A concurrent merge-on-read delete of id==4 commits first.
+ _, err := tbl.Delete(ctx, iceberg.EqualTo(iceberg.Reference("id"), int64(4)), nil)
+ require.NoError(t, err)
+
+ _, err = txn.Commit(ctx)
+ require.ErrorIs(t, err, table.ErrConflictingDeleteFiles)
+ })
+ }
+ }
+ }
+ }
+}
+
+// TestCopyOnWriteConflict_ConcurrentAppendCommits is the control: a concurrent
+// commit that adds no deletes (an append outside the filter) must not trip the
+// validator, so the replayed copy-on-write commit lands with both changes.
+func TestCopyOnWriteConflict_ConcurrentAppendCommits(t *testing.T) {
Review Comment:
Good catch, thanks. Added `TestCopyOnWriteConflict_ConcurrentDeleteOnUntouchedFileCommits` as you suggested. With your reject-every-delete mutation in place, all 4 new cases fail and the existing 24 still pass. With the scoped validator, all 28 pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): reject copy-on-write commits that would drop concurrent deletes [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #2099:
URL: https://github.com/apache/iceberg-go/pull/2099
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): reject copy-on-write commits that would drop concurrent deletes [iceberg-go]
Posted by "Revanth14 (via GitHub)" <gi...@apache.org>.
Revanth14 commented on PR #2099:
URL: https://github.com/apache/iceberg-go/pull/2099#issuecomment-6030240557
Thanks @zeroshade and @laskoviymishka for the reviews! I've addressed the remaining threads in the follow-up #2140:
- end-to-end test pinning the `ErrCommitDiverged` abort when a concurrent commit already removed the same file
- `slices.Concat` + `rewriteValidator(removed)`
- neutral wording for the conflict error messages
- releasing the test `TableReader`
The equality-delete branch is already covered by `TestValidateNoNewDeletesForRewrittenFiles_EqDeleteAlwaysConflicts`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): reject copy-on-write commits that would drop concurrent deletes [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2099:
URL: https://github.com/apache/iceberg-go/pull/2099#discussion_r4192871233
##########
table/cow_delete_conflict_test.go:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package table_test
+
+import (
+ "context"
+ "fmt"
+ "path/filepath"
+ "testing"
+
+ "github.com/apache/arrow-go/v18/arrow"
+ "github.com/apache/arrow-go/v18/arrow/array"
+ "github.com/apache/arrow-go/v18/arrow/memory"
+ "github.com/apache/iceberg-go"
+ iceio "github.com/apache/iceberg-go/io"
+ "github.com/apache/iceberg-go/table"
+ "github.com/stretchr/testify/require"
+)
+
+// newCoWConflictTestTable builds a table that deletes via merge-on-read (so a
+// concurrent delete writes delete files rather than rewriting data) and
+// retries commits, so a lost CAS race is refreshed and replayed. Both the
+// delete and update isolation levels are set to isolation.
+func newCoWConflictTestTable(t *testing.T, formatVersion string, isolation table.IsolationLevel) *table.Table {
+ t.Helper()
+
+ location := filepath.ToSlash(t.TempDir())
+ schema := iceberg.NewSchema(0,
+ iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+ iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+ )
+ meta, err := table.NewMetadata(schema, iceberg.UnpartitionedSpec, table.UnsortedSortOrder, location,
+ iceberg.Properties{
+ table.PropertyFormatVersion: formatVersion,
+ table.WriteDeleteModeKey: table.WriteModeMergeOnRead,
+ table.WriteDeleteIsolationLevelKey: string(isolation),
+ table.WriteUpdateIsolationLevelKey: string(isolation),
+ table.CommitNumRetriesKey: "2",
+ table.CommitMinRetryWaitMsKey: "1",
+ table.CommitMaxRetryWaitMsKey: "2",
+ table.CommitTotalRetryTimeoutMsKey: "1000",
+ })
+ require.NoError(t, err)
+
+ metaLoc := location + "/metadata/v1.metadata.json"
+ fsF := func(context.Context) (iceio.IO, error) { return iceio.LocalFS{}, nil }
+ cat := &concurrentTestCatalog{metadata: meta, location: metaLoc, fsF: fsF}
+
+ return table.New(table.Identifier{"db", "cow_delete_conflict"}, meta, metaLoc, fsF, cat)
+}
+
+func cowTestRecords(t *testing.T, rowsJSON string) array.RecordReader {
+ t.Helper()
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {Name: "id", Type: arrow.PrimitiveTypes.Int64, Nullable: false},
+ {Name: "data", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+ data, err := array.TableFromJSON(memory.DefaultAllocator, arrowSchema, []string{rowsJSON})
+ require.NoError(t, err)
+ t.Cleanup(data.Release)
+
+ return array.NewTableReader(data, -1)
+}
+
+// stageCopyOnWrite stages, without committing, a copy-on-write removal of the
+// rows matching filter through either Delete or Overwrite (with no new rows).
+func stageCopyOnWrite(t *testing.T, tbl *table.Table, op string, filter iceberg.BooleanExpression) *table.Transaction {
+ t.Helper()
+ ctx := context.Background()
+
+ txn := tbl.NewTransaction()
+ switch op {
+ case "delete":
+ require.NoError(t, txn.SetProperties(iceberg.Properties{
+ table.WriteDeleteModeKey: table.WriteModeCopyOnWrite,
+ }))
+ require.NoError(t, txn.Delete(ctx, filter, nil))
+ case "overwrite":
+ require.NoError(t, txn.Overwrite(ctx, cowTestRecords(t, `[]`), nil, table.WithOverwriteFilter(filter)))
+ default:
+ t.Fatalf("unknown op %q", op)
+ }
+
+ return txn
+}
+
+var (
+ cowFormatVersions = []string{"2", "3"}
+ cowIsolations = []table.IsolationLevel{table.IsolationSerializable, table.IsolationSnapshot}
+ cowOps = []string{"delete", "overwrite"}
+)
+
+// TestCopyOnWriteConflict_ConcurrentDeleteOnRemovedFile proves a copy-on-write
+// Delete or Overwrite is rejected, at every isolation level, when a concurrent
+// commit added a delete against a data file it removes. Without the validator,
+// refresh-and-replay swaps the original file for a rewrite built from the stale
+// snapshot, dropping the concurrent delete and resurrecting its row (#2090).
+func TestCopyOnWriteConflict_ConcurrentDeleteOnRemovedFile(t *testing.T) {
+ removals := []struct {
+ name string
+ filter iceberg.BooleanExpression
+ }{
+ // id==2 matches part of the file, so it is rewritten.
+ {"rewritten", iceberg.EqualTo(iceberg.Reference("id"), int64(2))},
+ // id>=1 matches every row, so the file is dropped outright.
+ {"fully deleted", iceberg.GreaterThanEqual(iceberg.Reference("id"), int64(1))},
+ }
+
+ for _, version := range cowFormatVersions {
+ for _, isolation := range cowIsolations {
+ for _, op := range cowOps {
+ for _, removal := range removals {
+ name := fmt.Sprintf("v%s/%s/%s/%s", version, isolation, op, removal.name)
+ t.Run(name, func(t *testing.T) {
+ ctx := context.Background()
+ tbl := appendTenRows(t, newCoWConflictTestTable(t, version, isolation))
+
+ txn := stageCopyOnWrite(t, tbl, op, removal.filter)
+
+ // A concurrent merge-on-read delete of id==4 commits first.
+ _, err := tbl.Delete(ctx, iceberg.EqualTo(iceberg.Reference("id"), int64(4)), nil)
Review Comment:
The concurrent writer here only ever stages a merge-on-read delete, so on v2 that's a position delete and on v3 a DV; the validator's equality-delete branch never fires in any of these runs. If we can stage a concurrent eq-delete writer, a case there would cover the one path the matrix misses; if iceberg-go can't write eq-deletes yet, calling that branch out as untested is fine.
##########
table/cow_delete_conflict_test.go:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package table_test
+
+import (
+ "context"
+ "fmt"
+ "path/filepath"
+ "testing"
+
+ "github.com/apache/arrow-go/v18/arrow"
+ "github.com/apache/arrow-go/v18/arrow/array"
+ "github.com/apache/arrow-go/v18/arrow/memory"
+ "github.com/apache/iceberg-go"
+ iceio "github.com/apache/iceberg-go/io"
+ "github.com/apache/iceberg-go/table"
+ "github.com/stretchr/testify/require"
+)
+
+// newCoWConflictTestTable builds a table that deletes via merge-on-read (so a
+// concurrent delete writes delete files rather than rewriting data) and
+// retries commits, so a lost CAS race is refreshed and replayed. Both the
+// delete and update isolation levels are set to isolation.
+func newCoWConflictTestTable(t *testing.T, formatVersion string, isolation table.IsolationLevel) *table.Table {
+ t.Helper()
+
+ location := filepath.ToSlash(t.TempDir())
+ schema := iceberg.NewSchema(0,
+ iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+ iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+ )
+ meta, err := table.NewMetadata(schema, iceberg.UnpartitionedSpec, table.UnsortedSortOrder, location,
+ iceberg.Properties{
+ table.PropertyFormatVersion: formatVersion,
+ table.WriteDeleteModeKey: table.WriteModeMergeOnRead,
+ table.WriteDeleteIsolationLevelKey: string(isolation),
+ table.WriteUpdateIsolationLevelKey: string(isolation),
+ table.CommitNumRetriesKey: "2",
+ table.CommitMinRetryWaitMsKey: "1",
+ table.CommitMaxRetryWaitMsKey: "2",
+ table.CommitTotalRetryTimeoutMsKey: "1000",
+ })
+ require.NoError(t, err)
+
+ metaLoc := location + "/metadata/v1.metadata.json"
+ fsF := func(context.Context) (iceio.IO, error) { return iceio.LocalFS{}, nil }
+ cat := &concurrentTestCatalog{metadata: meta, location: metaLoc, fsF: fsF}
+
+ return table.New(table.Identifier{"db", "cow_delete_conflict"}, meta, metaLoc, fsF, cat)
+}
+
+func cowTestRecords(t *testing.T, rowsJSON string) array.RecordReader {
+ t.Helper()
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {Name: "id", Type: arrow.PrimitiveTypes.Int64, Nullable: false},
+ {Name: "data", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+ data, err := array.TableFromJSON(memory.DefaultAllocator, arrowSchema, []string{rowsJSON})
+ require.NoError(t, err)
+ t.Cleanup(data.Release)
+
+ return array.NewTableReader(data, -1)
Review Comment:
`array.NewTableReader` retains `data`, but only `data.Release` is registered for cleanup, so the reader itself is never released. Harmless under the default allocator, but it leaks a ref under a `CheckedAllocator`. I'd create the reader first and register `t.Cleanup(rdr.Release)` alongside the table release, unless the `Append`/`Overwrite` callers already release it.
##########
table/transaction.go:
##########
@@ -2478,6 +2478,20 @@ func (t *Transaction) performCopyOnWriteDeletion(ctx context.Context, operation
}
}
+ // Reject the commit if a concurrent snapshot added deletes against any
+ // data file this operation removes. The overwriteFiles producer only
+ // checks added data files (and only under serializable isolation);
+ // without this check a refresh-and-replay would swap the original file
+ // for a rewrite built from the stale snapshot, dropping the concurrent
+ // deletes and resurrecting their rows. Mirrors Java's copy-on-write
+ // validateNoConflictingDeletes: no isolation gating.
+ removed := append(slices.Clip(filesToDelete), filesToRewrite...)
Review Comment:
The `slices.Clip` plus `append` is correct: clipping caps `filesToDelete` so the append forces a fresh backing array and we never write into its spare capacity. It reads as non-obvious enough that I'd either add a one-line comment on why the clip is load-bearing, or switch to `slices.Concat(filesToDelete, filesToRewrite)` (1.22+), which says the same thing without the idiom.
##########
table/transaction.go:
##########
@@ -2478,6 +2478,20 @@ func (t *Transaction) performCopyOnWriteDeletion(ctx context.Context, operation
}
}
+ // Reject the commit if a concurrent snapshot added deletes against any
+ // data file this operation removes. The overwriteFiles producer only
+ // checks added data files (and only under serializable isolation);
+ // without this check a refresh-and-replay would swap the original file
+ // for a rewrite built from the stale snapshot, dropping the concurrent
+ // deletes and resurrecting their rows. Mirrors Java's copy-on-write
+ // validateNoConflictingDeletes: no isolation gating.
+ removed := append(slices.Clip(filesToDelete), filesToRewrite...)
+ if len(removed) > 0 {
+ t.addValidator(func(cc *conflictContext) error {
Review Comment:
We register the new-deletes half of the check here but not the deleted-data-files half. `performMergeOnReadDeletion` also registers `validateDataFilesExist`, and Java's `validateNoConflictingDeletes` rejects when a concurrent commit removed or rewrote a file this operation is also removing. Without that, if a concurrent CoW delete or compaction rewrites one of these files first, the replay re-removes a file that's already gone and rebuilds it from the stale snapshot. I'd confirm `overwriteFiles` already rejects that on its own; if it doesn't, I'd add `validateDataFilesExist(cc, removed)` next to this registration or track it as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): reject copy-on-write commits that would drop concurrent deletes [iceberg-go]
Posted by "Revanth14 (via GitHub)" <gi...@apache.org>.
Revanth14 commented on PR #2099:
URL: https://github.com/apache/iceberg-go/pull/2099#issuecomment-5966842887
> Registering the existing `validateNoNewDeletesForRewrittenFiles` through `addValidator` for every data file the copy-on-write path removes closes #2090 and matches Java's copy-on-write `validateNoConflictingDeletes()` at both isolation levels. I confirmed that the new conflict cases fail without the fix (the replay brings `id=4` back) and pass here.
>
> ### Smaller observations
> * The append control can't catch an over-broad validator; see the inline comment for a v3 case that can.
> * On v2, the partition fallback fires for every position delete iceberg-go itself writes. `referenced_data_file` is unset, and the default `truncate(16)` metrics clip the `file_path` bounds. So a concurrent MoR delete on a _different_ file in the same partition (anywhere, if the table is unpartitioned) also rejects a retried CoW commit; a probe confirmed this. It's safe and your description mentions the fallback. Java's writer, though, keeps exact `file_path` bounds (`MetricsConfig.forPositionDelete()`), so its fallback rarely fires. That's worth a writer-side follow-up rather than a change here.
> * Heads-up: #2046 turns `filesToRewrite` into `[]iceberg.ManifestEntry`, so whichever PR lands second needs to adapt the `removed` slice.
> * The commit is missing the `Signed-off-by` trailer that CONTRIBUTING.md asks for.
>
> > _This review was drafted by an AI-assisted tool and
> > confirmed by an Apache Iceberg maintainer. The maintainer
> > approving this PR has read the findings and signed off. If
> > something feels off, please reply on the PR and a maintainer
> > will follow up._
> > _More on how Apache Iceberg handles maintainer review:_
> > [CONTRIBUTING.md](https://github.com/apache/iceberg-go/blob/main/CONTRIBUTING.md).
Thanks for the review!
applied the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): reject copy-on-write commits that would drop concurrent deletes [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #2099:
URL: https://github.com/apache/iceberg-go/pull/2099#discussion_r4168426737
##########
table/cow_delete_conflict_test.go:
##########
@@ -0,0 +1,172 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package table_test
+
+import (
+ "context"
+ "fmt"
+ "path/filepath"
+ "testing"
+
+ "github.com/apache/arrow-go/v18/arrow"
+ "github.com/apache/arrow-go/v18/arrow/array"
+ "github.com/apache/arrow-go/v18/arrow/memory"
+ "github.com/apache/iceberg-go"
+ iceio "github.com/apache/iceberg-go/io"
+ "github.com/apache/iceberg-go/table"
+ "github.com/stretchr/testify/require"
+)
+
+// newCoWConflictTestTable builds a table that deletes via merge-on-read (so a
+// concurrent delete writes delete files rather than rewriting data) and
+// retries commits, so a lost CAS race is refreshed and replayed. Both the
+// delete and update isolation levels are set to isolation.
+func newCoWConflictTestTable(t *testing.T, formatVersion string, isolation table.IsolationLevel) *table.Table {
+ t.Helper()
+
+ location := filepath.ToSlash(t.TempDir())
+ schema := iceberg.NewSchema(0,
+ iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+ iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+ )
+ meta, err := table.NewMetadata(schema, iceberg.UnpartitionedSpec, table.UnsortedSortOrder, location,
+ iceberg.Properties{
+ table.PropertyFormatVersion: formatVersion,
+ table.WriteDeleteModeKey: table.WriteModeMergeOnRead,
+ table.WriteDeleteIsolationLevelKey: string(isolation),
+ table.WriteUpdateIsolationLevelKey: string(isolation),
+ table.CommitNumRetriesKey: "2",
+ table.CommitMinRetryWaitMsKey: "1",
+ table.CommitMaxRetryWaitMsKey: "2",
+ table.CommitTotalRetryTimeoutMsKey: "1000",
+ })
+ require.NoError(t, err)
+
+ metaLoc := location + "/metadata/v1.metadata.json"
+ fsF := func(context.Context) (iceio.IO, error) { return iceio.LocalFS{}, nil }
+ cat := &concurrentTestCatalog{metadata: meta, location: metaLoc, fsF: fsF}
+
+ return table.New(table.Identifier{"db", "cow_delete_conflict"}, meta, metaLoc, fsF, cat)
+}
+
+func cowTestRecords(t *testing.T, rowsJSON string) array.RecordReader {
+ t.Helper()
+
+ arrowSchema := arrow.NewSchema([]arrow.Field{
+ {Name: "id", Type: arrow.PrimitiveTypes.Int64, Nullable: false},
+ {Name: "data", Type: arrow.BinaryTypes.String, Nullable: true},
+ }, nil)
+ data, err := array.TableFromJSON(memory.DefaultAllocator, arrowSchema, []string{rowsJSON})
+ require.NoError(t, err)
+ t.Cleanup(data.Release)
+
+ return array.NewTableReader(data, -1)
+}
+
+// stageCopyOnWrite stages, without committing, a copy-on-write removal of the
+// rows matching filter through either Delete or Overwrite (with no new rows).
+func stageCopyOnWrite(t *testing.T, tbl *table.Table, op string, filter iceberg.BooleanExpression) *table.Transaction {
+ t.Helper()
+ ctx := context.Background()
+
+ txn := tbl.NewTransaction()
+ switch op {
+ case "delete":
+ require.NoError(t, txn.SetProperties(iceberg.Properties{
+ table.WriteDeleteModeKey: table.WriteModeCopyOnWrite,
+ }))
+ require.NoError(t, txn.Delete(ctx, filter, nil))
+ case "overwrite":
+ require.NoError(t, txn.Overwrite(ctx, cowTestRecords(t, `[]`), nil, table.WithOverwriteFilter(filter)))
+ default:
+ t.Fatalf("unknown op %q", op)
+ }
+
+ return txn
+}
+
+var (
+ cowFormatVersions = []string{"2", "3"}
+ cowIsolations = []table.IsolationLevel{table.IsolationSerializable, table.IsolationSnapshot}
+ cowOps = []string{"delete", "overwrite"}
+)
+
+// TestCopyOnWriteConflict_ConcurrentDeleteOnRemovedFile proves a copy-on-write
+// Delete or Overwrite is rejected, at every isolation level, when a concurrent
+// commit added a delete against a data file it removes. Without the validator,
+// refresh-and-replay swaps the original file for a rewrite built from the stale
+// snapshot, dropping the concurrent delete and resurrecting its row (#2090).
+func TestCopyOnWriteConflict_ConcurrentDeleteOnRemovedFile(t *testing.T) {
+ removals := []struct {
+ name string
+ filter iceberg.BooleanExpression
+ }{
+ // id==2 matches part of the file, so it is rewritten.
+ {"rewritten", iceberg.EqualTo(iceberg.Reference("id"), int64(2))},
+ // id>=1 matches every row, so the file is dropped outright.
+ {"fully deleted", iceberg.GreaterThanEqual(iceberg.Reference("id"), int64(1))},
+ }
+
+ for _, version := range cowFormatVersions {
+ for _, isolation := range cowIsolations {
+ for _, op := range cowOps {
+ for _, removal := range removals {
+ name := fmt.Sprintf("v%s/%s/%s/%s", version, isolation, op, removal.name)
+ t.Run(name, func(t *testing.T) {
+ ctx := context.Background()
+ tbl := appendTenRows(t, newCoWConflictTestTable(t, version, isolation))
+
+ txn := stageCopyOnWrite(t, tbl, op, removal.filter)
+
+ // A concurrent merge-on-read delete of id==4 commits first.
+ _, err := tbl.Delete(ctx, iceberg.EqualTo(iceberg.Reference("id"), int64(4)), nil)
+ require.NoError(t, err)
+
+ _, err = txn.Commit(ctx)
+ require.ErrorIs(t, err, table.ErrConflictingDeleteFiles)
+ })
+ }
+ }
+ }
+ }
+}
+
+// TestCopyOnWriteConflict_ConcurrentAppendCommits is the control: a concurrent
+// commit that adds no deletes (an append outside the filter) must not trip the
+// validator, so the replayed copy-on-write commit lands with both changes.
+func TestCopyOnWriteConflict_ConcurrentAppendCommits(t *testing.T) {
Review Comment:
This control can't tell the scoped validator apart from one that rejects *every* concurrent delete, because an append adds no delete files. I swapped the closure for `cc.forEachAddedEntry(iceberg.ManifestContentDeletes, …return ErrConflictingDeleteFiles)` and all 24 subtests still passed.
A v3 case where the concurrent delete hits a file this commit leaves alone would catch that. DVs carry `referenced_data_file`; v2 deletes here are partition-scoped, so they conflict anyway. This passes at head and fails under that mutation:
~~~go
func TestCopyOnWriteConflict_ConcurrentDeleteOnUntouchedFileCommits(t *testing.T) {
for _, isolation := range cowIsolations {
for _, op := range cowOps {
t.Run(fmt.Sprintf("v3/%s/%s", isolation, op), func(t *testing.T) {
ctx := context.Background()
tbl := appendTenRows(t, newCoWConflictTestTable(t, "3", isolation))
// A second data file that the id==2 removal does not touch.
tbl, err := tbl.Append(ctx, cowTestRecords(t, `[{"id":11,"data":"k"},{"id":12,"data":"l"}]`), nil)
require.NoError(t, err)
txn := stageCopyOnWrite(t, tbl, op, iceberg.EqualTo(iceberg.Reference("id"), int64(2)))
_, err = tbl.Delete(ctx, iceberg.EqualTo(iceberg.Reference("id"), int64(12)), nil)
require.NoError(t, err)
committed, err := txn.Commit(ctx)
require.NoError(t, err)
require.Equal(t, []int64{1, 3, 4, 5, 6, 7, 8, 9, 10, 11}, idsInTable(t, committed))
})
}
}
}
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org