You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mzzz-zzm (via GitHub)" <gi...@apache.org> on 2026/05/04 08:12:27 UTC

[I] bug(table): Conflict detection is silently bypassed when the writer starts from an empty table [iceberg-go]

mzzz-zzm opened a new issue, #977:
URL: https://github.com/apache/iceberg-go/issues/977

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   When a writer loads a table that has **no snapshot yet** on the target branch
   (empty table, or a branch that the writer is about to create), the
   `newConflictContext` helper returns an **empty** `concurrent` list — it treats
   the situation as "no concurrent activity". If another writer then commits a
   snapshot between the first writer's load time and its commit time, the conflict
   validators never see those concurrent commits. A `RowDelta` carrying
   equality-delete files will be accepted even though it should be rejected under
   `SERIALIZABLE` isolation.
   
   ## Affected component
   
   `table/conflict_validation.go` — `newConflictContext()`
   
   ## Steps to reproduce
   
   The test is self-contained and lives in **package `table`** (internal test) so
   it can call the unexported `newConflictContext` directly.  The helpers it uses
   (`newConflictTestMetadata`, `MainBranch`) are already in
   `table/conflict_validation_test.go` on `main`.
   
   Save the file below as `table/bug_repro_empty_table_conflict_test.go` in an
   unmodified checkout of `main` and run:
   
   ```
   go test ./table/ -run TestBugRepro_EmptyTableConflictDetectionBypassed -v
   ```
   
   Expected output on unfixed upstream:
   
   ```
   --- FAIL: TestBugRepro_EmptyTableConflictDetectionBypassed
       bug_repro_empty_table_conflict_test.go:76:
           Error:    "[]" should have 1 item(s), but has 0
           Messages: snapshot 42 must appear in concurrent when base has no branch head
   ```
   
   Full source of `table/bug_repro_empty_table_conflict_test.go`:
   
   ```go
   package table
   
   import (
   	"testing"
   
   	"github.com/stretchr/testify/assert"
   	"github.com/stretchr/testify/require"
   )
   
   func TestBugRepro_EmptyTableConflictDetectionBypassed(t *testing.T) {
   	// Writer loaded the table before any snapshot was committed on main.
   	base := newConflictTestMetadata(t, nil)
   
   	// Meanwhile another writer committed snapshot 42.
   	head := int64(42)
   	current := newConflictTestMetadata(t, &head)
   
   	ctx, err := newConflictContext(base, current, MainBranch, nil, true)
   	require.NoError(t, err)
   
   	// Snapshot 42 was committed while Writer had no view of the branch.
   	// It MUST appear in ctx.concurrent so that conflict validators (e.g.
   	// RowDelta's serializable check) can inspect the data files it added.
   	//
   	// BUG: on upstream, ctx.concurrent is empty — snapshot 42 is invisible.
   	// Any equality-delete commit from the empty-base writer will be accepted
   	// without validating against the concurrent data.
   	assert.Len(t, ctx.concurrent, 1,
   		"snapshot 42 must appear in concurrent when base has no branch head")
   }
   ```
   
   ## Root cause
   
   The relevant section of `table/conflict_validation.go` on `main`:
   
   ```go
   baseHead := base.SnapshotByName(branch)
   if baseHead == nil {
       // Writer has no view of this branch yet (e.g. creating it) —
       // by definition there are no concurrent commits to validate against.
       return &conflictContext{current: current, branch: branch, fs: fs, caseSensitive: caseSensitive}, nil
   }
   ```
   
   When `baseHead` is `nil` the code returns immediately with an empty
   `concurrent` list, reasoning that there was "nothing to compare". But a nil
   `baseHead` simply means the writer did not see any snapshot when it loaded the
   table. Snapshots committed *after* that load but *before* this commit are still
   real concurrent activity that validators must examine.
   
   ## Concrete scenario where this matters
   
   1. Table is empty. Worker A begins building an equality-delete commit
      (e.g. a streaming DELETE job that marks rows as deleted by field value).
   2. Worker B appends new data files to the same partition and commits first.
      The table now has snapshot B.
   3. Worker A calls `Commit`. Under `SERIALIZABLE` isolation, Worker A's
      equality-delete must be rejected: Worker B added data that Worker A's
      delete predicate may not cover — Worker A must reload and rebuild.
   4. **Bug:** `newConflictContext` sees `baseHead == nil` and returns zero
      concurrent snapshots. The `RowDelta` validator never checks Worker B's
      data files. Worker A's commit is accepted and the equality deletes are
      applied against data that the writer never saw.
   
   ## Expected behaviour
   
   When a writer started from an empty branch (no prior snapshot), all snapshots
   currently on that branch were committed after the writer loaded the table and
   are therefore concurrent. The returned `conflictContext.concurrent` list must
   contain all of them so validators can decide whether a conflict exists.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] bug(table): Conflict detection is silently bypassed when the writer starts from an empty table [iceberg-go]

Posted by "mzzz-zzm (via GitHub)" <gi...@apache.org>.
mzzz-zzm commented on issue #977:
URL: https://github.com/apache/iceberg-go/issues/977#issuecomment-4369969986

   ## Suggested fix
   
   When `newConflictContext` is called with a `base` that has no snapshot on the target branch (`base.SnapshotByName(branch) == nil`), the function returned an empty `concurrent` list. The fix below uses `AncestorsOf` to populate `concurrent` with all snapshots currently on the branch — those are all commits that happened after the writer loaded the (then-empty) table, and they must be inspected by validators.
   
   This change touches **one function in one file** and is independent of the manifest list rebuild fix and the RowDelta partition filter fix.
   
   ## File changed
   
   `table/conflict_validation.go` — `newConflictContext()`
   
   ## Diff
   
   ```diff
    baseHead := base.SnapshotByName(branch)
    if baseHead == nil {
   -	// Writer has no view of this branch yet (e.g. creating it) —
   -	// by definition there are no concurrent commits to validate
   -	// against.
   -	return &conflictContext{current: current, branch: branch, fs: fs, caseSensitive: caseSensitive}, nil
   +	// Writer started from a table that had no snapshot on this branch
   +	// (empty table, or branch not yet created by the writer). All
   +	// snapshots now present on the branch were committed concurrently
   +	// and must be examined by validators that care about concurrent
   +	// writes (e.g. RowDelta with equality deletes under SERIALIZABLE).
   +	// Fast-append validators safely short-circuit on non-empty
   +	// concurrent lists, so this does not regress the common path.
   +	allConcurrent := AncestorsOf(currentHead.SnapshotID, current.SnapshotByID)
   +	return &conflictContext{
   +		current:       current,
   +		branch:        branch,
   +		fs:            fs,
   +		caseSensitive: caseSensitive,
   +		concurrent:    allConcurrent,
   +	}, nil
    }
   ```
   
   ## Why this is safe for fast-append
   
   Fast-append and merge-append producers do not register a validator (their `producerImpl.validate` is a no-op). A non-empty `concurrent` list is walked only when a validator calls `forEachAddedEntry` or `validateNoConflictingDataFiles`. Writers that do not register validators pay only the cost of the `AncestorsOf` walk (proportional to the number of snapshots on the branch) with no change in commit acceptance behaviour.
   
   ## Existing test that encodes the old (wrong) behaviour
   
   `TestNewConflictContext_WriterHasNoBranchView` in `table/conflict_validation_test.go` currently asserts `assert.Empty(t, ctx.concurrent)`. That assertion must be updated to reflect the correct behaviour:
   
   ```diff
   -assert.Empty(t, ctx.concurrent)
   +assert.Len(t, ctx.concurrent, 1)
   +assert.Equal(t, int64(7), ctx.concurrent[0].SnapshotID)
   ```
   
   ## Test
   
   Run the reproducer from the bug report:
   
   ```
   go test ./table/ -run TestBugRepro_EmptyTableConflictDetectionBypassed -v
   ```
   
   Expected after this fix:
   
   ```
   --- PASS: TestBugRepro_EmptyTableConflictDetectionBypassed (0.00s)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] bug(table): Conflict detection is silently bypassed when the writer starts from an empty table [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #977: bug(table): Conflict detection is silently bypassed when the writer starts from an empty table
URL: https://github.com/apache/iceberg-go/issues/977


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org