You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/09/25 06:12:42 UTC
[PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
1fanwang opened a new pull request, #2057:
URL: https://github.com/apache/iceberg-go/pull/2057
Manifest merging can fail a valid commit when a merge bin contains only historical `DELETED` entries. Those entries are intentionally filtered out, but the merge path opened a new manifest writer first, then closed it empty and returned `empty manifest file has been written`.
After this change, merge bins that retain no entries produce no replacement manifest. Current-snapshot deletes are still copied, non-empty bins still merge normally, and empty `ManifestWriter` calls still fail for direct callers.
Closes https://github.com/apache/iceberg-go/issues/2037
## Testing
<details><summary>Raw logs</summary>
Before the fix:
```text
$ go test ./table -run 'TestManifestMerge(SkipsHistoricalDeletedOnlyManifest|KeepsCurrentSnapshotDeletedEntries|GroupDropsEmptyMergedBin)'
--- FAIL: TestManifestMergeSkipsHistoricalDeletedOnlyManifest (0.00s)
snapshot_producers_test.go:601:
Error: Received unexpected error:
empty manifest file has been written
--- FAIL: TestManifestMergeGroupDropsEmptyMergedBin (0.00s)
snapshot_producers_test.go:664:
Error: Received unexpected error:
empty manifest file has been written
FAIL
FAIL github.com/apache/iceberg-go/table 0.498s
FAIL
```
After the fix:
```text
$ go test ./table -run 'TestManifestMerge(SkipsHistoricalDeletedOnlyManifest|KeepsCurrentSnapshotDeletedEntries|GroupDropsEmptyMergedBin)'
ok github.com/apache/iceberg-go/table 0.543s
$ go test ./table/...
ok github.com/apache/iceberg-go/table 8.299s
ok github.com/apache/iceberg-go/table/compaction 2.538s
ok github.com/apache/iceberg-go/table/dv (cached)
ok github.com/apache/iceberg-go/table/internal (cached)
ok github.com/apache/iceberg-go/table/substrait (cached)
$ go vet ./table/...
$ golangci-lint run --timeout=10m ./table/...
0 issues.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057#discussion_r4149598173
##########
table/snapshot_producers.go:
##########
@@ -507,18 +507,32 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
return nil, err
}
- wr, path, counter, fileCloser, err := m.snap.newManifestWriter(spec)
- if err != nil {
- return nil, err
- }
- defer internal.CheckedClose(fileCloser, &err)
+ var wr *iceberg.ManifestWriter
+ var path string
+ var counter *internal.CountingWriter
+ var fileCloser io.Closer
writerClosed := false
+ // Close the ManifestWriter before the underlying file so a mid-loop
+ // write failure flushes into an open file, not a closed one.
defer func() {
- if !writerClosed {
+ if wr != nil && !writerClosed {
internal.CheckedClose(wr, &err)
}
+ if fileCloser != nil {
+ internal.CheckedClose(fileCloser, &err)
+ }
}()
+ ensureWriter := func() error {
+ if wr != nil {
+ return nil
+ }
+
+ wr, path, counter, fileCloser, err = m.snap.newManifestWriter(spec)
Review Comment:
Still here from last round: `ensureWriter` writes into the outer named-return `err` and returns the same value, so every call site double-assigns it. It's harmless today, but it's a footgun: a call site switching to `if err := ensureWriter()` would shadow locally and silently leak the error into the named return and the deferred closes. I'd have the closure use a local and return it:
```go
ensureWriter := func() error {
if wr != nil {
return nil
}
var err error
wr, path, counter, fileCloser, err = m.snap.newManifestWriter(spec)
return err
}
```
##########
table/snapshot_producers_test.go:
##########
@@ -578,6 +578,130 @@ func TestManifestMergeManagerClosesWriterOnError(t *testing.T) {
require.ErrorIs(t, err, errLimitedWrite)
}
+func TestManifestMergeManagerClosesWriterBeforeFileOnWriteFailure(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+
+ // Use a byte-limited IO that fails after the writer is opened and
+ // has written some data, but before all entries are processed.
+ mem := newMemIO(manifestHeaderSize(t, 2, spec, schema), errLimitedWrite)
Review Comment:
This test can't fail on the ordering it's named for. `limitedWriteCloser` has no write-after-close detection; its `Write` only checks the byte budget, so if `fileCloser` closed before `wr`, the `wr.Close()` flush would still just return `errLimitedWrite`, and the final "assertion" (a prose comment, not a `require`) passes either way. That leaves the round-1 blocker guarded by a test that can't catch a regression.
I'd switch to `trackingIO`/`trackingWriteCloser`, fail the write after the header lands, and assert what `TestCommitManifestsCloseFailureReturnsNoUpdates` already does:
```go
require.NotContains(t, err.Error(), "write after close",
"manifest writer must be closed before its underlying file closer")
```
That's the piece that actually locks the ordering down.
##########
table/snapshot_producers.go:
##########
@@ -543,6 +566,13 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
}
}
+ // A bin with no live entries produces no manifest. This diverges from
+ // Java/PyIceberg, which write a zero-count manifest; the omission is
Review Comment:
Small thing on the new comment: Java does write the zero-count manifest, but PyIceberg has merge disabled by default (`MANIFEST_MERGE_ENABLED_DEFAULT = False`), so I couldn't confirm its merge path does the same. I'd narrow this to "the Java reference implementation" rather than "Java/PyIceberg" unless someone's actually traced PyIceberg's merge.
##########
table/snapshot_producers.go:
##########
@@ -1762,16 +1795,13 @@ func (sp *snapshotProducer) commitManifests(newManifests, addedContent []iceberg
// creates it).
baseHeadID := sp.txn.baseRefSnapshotID(branch)
- return []Update{
- addSnap,
- // Carry over the branch's existing retention settings so advancing
- // the ref on commit does not silently discard them. The update
- // encodes exactly the current ref's retention (settings the branch
- // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
- // applies a set-snapshot-ref as a pure replace, so this fully
- // determines the resulting ref rather than merging with the old one.
- sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef),
- }, []Requirement{
- AssertRefSnapshotID(branch, baseHeadID),
- }, nil
+ // Carry over the branch's existing retention settings so advancing
+ // the ref on commit does not silently discard them. The update
+ // encodes exactly the current ref's retention (settings the branch
+ // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
+ // applies a set-snapshot-ref as a pure replace, so this fully
+ // determines the resulting ref rather than merging with the old one.
+ retainingSnapshotRef := sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef)
Review Comment:
Same ask as last round: the `commitManifests` reflow (extracting `retainingSnapshotRef`) and the trailing-comma reformats aren't part of the fix. I'd split them into a separate tidy commit so the bug-fix diff stays focused; the trailing commas may be `gofumpt`, but they still belong on their own.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057#discussion_r4102962036
##########
table/snapshot_producers.go:
##########
@@ -507,17 +507,31 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
return nil, err
}
- wr, path, counter, fileCloser, err := m.snap.newManifestWriter(spec)
- if err != nil {
- return nil, err
- }
- defer internal.CheckedClose(fileCloser, &err)
+ var wr *iceberg.ManifestWriter
+ var path string
+ var counter *internal.CountingWriter
+ var fileCloser io.Closer
writerClosed := false
defer func() {
- if !writerClosed {
+ if wr != nil && !writerClosed {
internal.CheckedClose(wr, &err)
}
}()
+ defer func() {
+ if fileCloser != nil {
+ internal.CheckedClose(fileCloser, &err)
+ }
+ }()
+
+ ensureWriter := func() error {
+ if wr != nil {
+ return nil
+ }
+
+ wr, path, counter, fileCloser, err = m.snap.newManifestWriter(spec)
Review Comment:
This assigns into the named-return `err`, but every call site checks the closure's return value inside a `for entry, err := range ...` where the `:=` shadows the outer `err`, so the explicit `return nil, err` is what actually propagates and this write to the outer `err` is dead. It reads like the named return is tracked automatically, which invites a future edit to drop the explicit check on the false assumption `err` is already set. I'd give the closure its own local:
```go
ensureWriter := func() error {
if wr != nil {
return nil
}
var werr error
wr, path, counter, fileCloser, werr = m.snap.newManifestWriter(spec)
return werr
}
```
##########
table/snapshot_producers.go:
##########
@@ -577,7 +604,9 @@ func (m *manifestMergeManager) mergeGroup(firstManifest iceberg.ManifestFile, sp
if err != nil {
return nil, err
}
- output = append(output, created)
+ if created != nil {
Review Comment:
Separate from this fix but related: the `len(bin)==1` short-circuit in `mergeGroup` passes a lone manifest through unfiltered, so a manifest that's 100% historical-DELETED and lands alone in its bin (already at target size, or gated by `minCountToMerge`) never reaches `createManifest` and carries its dead tombstones forward indefinitely. This PR fixes the commit crash, not that; fine as a follow-up, but worth noting so #2037 isn't assumed fully closed.
##########
table/snapshot_producers.go:
##########
@@ -543,6 +566,10 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
}
}
+ if wr == nil {
Review Comment:
Java's `ManifestMergeManager` and PyIceberg both keep a zero-count manifest here rather than dropping it, so for identical merge history Go emits fewer manifest files than the other clients. That's the intended tradeoff from #2037 and it's read-safe, but it's a permanent divergence worth a short comment right here (and a line in the PR description) so nobody diffing manifest counts across clients treats it as a bug and "fixes" it back to parity later.
##########
table/snapshot_producers.go:
##########
@@ -507,17 +507,31 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
return nil, err
}
- wr, path, counter, fileCloser, err := m.snap.newManifestWriter(spec)
- if err != nil {
- return nil, err
- }
- defer internal.CheckedClose(fileCloser, &err)
+ var wr *iceberg.ManifestWriter
+ var path string
+ var counter *internal.CountingWriter
+ var fileCloser io.Closer
writerClosed := false
defer func() {
- if !writerClosed {
+ if wr != nil && !writerClosed {
internal.CheckedClose(wr, &err)
}
}()
+ defer func() {
Review Comment:
The two cleanup defers run in the wrong order on the error path: LIFO now closes `fileCloser` before `wr`, so if an entry write fails mid-loop after the writer's already open, `wr.Close()` flushes the `ManifestWriter` into an already-closed file. That's the "write after close" hazard `TestCommitManifestsCloseFailureReturnsNoUpdates` was written to guard, and every other writer-cleanup site in this file registers the fileCloser defer first so `wr` flushes first.
I'd collapse both into one closure with an explicit order rather than relying on LIFO across two statements:
```go
defer func() {
if wr != nil && !writerClosed {
internal.CheckedClose(wr, &err)
}
if fileCloser != nil {
internal.CheckedClose(fileCloser, &err)
}
}()
```
And add a regression test that opens the writer then fails a later entry write (the `trackingIO`/`failWriteAt` idiom), asserting `NotContains "write after close"`. None of the new tests hit that ordering today, which is why this slips through.
##########
table/snapshot_producers.go:
##########
@@ -1762,16 +1792,13 @@ func (sp *snapshotProducer) commitManifests(newManifests, addedContent []iceberg
// creates it).
baseHeadID := sp.txn.baseRefSnapshotID(branch)
- return []Update{
- addSnap,
- // Carry over the branch's existing retention settings so advancing
- // the ref on commit does not silently discard them. The update
- // encodes exactly the current ref's retention (settings the branch
- // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
- // applies a set-snapshot-ref as a pure replace, so this fully
- // determines the resulting ref rather than merging with the old one.
- sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef),
- }, []Requirement{
- AssertRefSnapshotID(branch, baseHeadID),
- }, nil
+ // Carry over the branch's existing retention settings so advancing
+ // the ref on commit does not silently discard them. The update
+ // encodes exactly the current ref's retention (settings the branch
+ // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
+ // applies a set-snapshot-ref as a pure replace, so this fully
+ // determines the resulting ref rather than merging with the old one.
+ retainingSnapshotRef := sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef)
Review Comment:
This `commitManifests` refactor (and the `NewDataFileBuilder` / `mergeConcurrency` reformats) are behavior-neutral and unrelated to the empty-manifest fix. I'd pull them out so blame and bisect stay clean on the actual change.
##########
table/snapshot_producers_test.go:
##########
@@ -578,6 +578,93 @@ func TestManifestMergeManagerClosesWriterOnError(t *testing.T) {
require.ErrorIs(t, err, errLimitedWrite)
}
+func TestManifestMergeSkipsHistoricalDeletedOnlyManifest(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+
+ oldSnapshotID := sp.snapshotID - 1
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ manifestFile := writeTestManifestWithContent(t, trackIO, spec, schema, oldSnapshotID,
+ "table-location/metadata/historical-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &oldSnapshotID, &sequenceNumber, nil, df),
+ })
+
+ trackIO.writers = make(map[string]*trackingWriteCloser)
+
+ mgr := manifestMergeManager{snap: sp}
+ created, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifestFile})
+ require.NoError(t, err)
+ require.Nil(t, created)
+ require.Zero(t, trackIO.GetWriterCount())
+}
+
+func TestManifestMergeKeepsCurrentSnapshotDeletedEntries(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ txn, wfs := createTestTransactionWithMemIO(t, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, wfs, nil, nil)
+
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ manifestFile := writeTestManifestWithContent(t, wfs, spec, schema, sp.snapshotID,
+ "mem://default/table-location/metadata/current-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &sp.snapshotID, &sequenceNumber, nil, df),
+ })
+
+ mgr := manifestMergeManager{snap: sp}
+ created, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifestFile})
+ require.NoError(t, err)
+ require.NotNil(t, created)
+
+ var entries []iceberg.ManifestEntry
+ for entry, err := range created.Entries(wfs, false) {
+ require.NoError(t, err)
+ entries = append(entries, entry)
+ }
+ require.Len(t, entries, 1)
+ require.Equal(t, iceberg.EntryStatusDELETED, entries[0].Status())
+ require.Equal(t, sp.snapshotID, entries[0].SnapshotID())
+ require.Equal(t, df.FilePath(), entries[0].DataFile().FilePath())
+}
+
+func TestManifestMergeGroupDropsEmptyMergedBin(t *testing.T) {
Review Comment:
The three new tests cover the happy paths well, but #2037's validation list explicitly calls for "a read error occurring before an output writer is created," and none of these hit it. I'd add a two-manifest bin where the first is all-historical-delete (writer stays `nil`) and the second fails to read, and assert `createManifest` returns the read error, not `(nil, nil)`, and that no writer was opened. That's exactly the `ensureWriter`-gating seam most likely to regress if the error check is ever reordered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057#discussion_r4162556064
##########
table/snapshot_producers.go:
##########
@@ -543,6 +566,13 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
}
}
+ // A bin with no live entries produces no manifest. This diverges from
+ // Java/PyIceberg, which write a zero-count manifest; the omission is
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/4e958128577cf5ba785051ae0e3b6b09f481b8ef. The comment now names only the Java reference implementation.
##########
table/snapshot_producers.go:
##########
@@ -1762,16 +1795,13 @@ func (sp *snapshotProducer) commitManifests(newManifests, addedContent []iceberg
// creates it).
baseHeadID := sp.txn.baseRefSnapshotID(branch)
- return []Update{
- addSnap,
- // Carry over the branch's existing retention settings so advancing
- // the ref on commit does not silently discard them. The update
- // encodes exactly the current ref's retention (settings the branch
- // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
- // applies a set-snapshot-ref as a pure replace, so this fully
- // determines the resulting ref rather than merging with the old one.
- sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef),
- }, []Requirement{
- AssertRefSnapshotID(branch, baseHeadID),
- }, nil
+ // Carry over the branch's existing retention settings so advancing
+ // the ref on commit does not silently discard them. The update
+ // encodes exactly the current ref's retention (settings the branch
+ // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
+ // applies a set-snapshot-ref as a pure replace, so this fully
+ // determines the resulting ref rather than merging with the old one.
+ retainingSnapshotRef := sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef)
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/dcc96bbbaa851bbfc2e66eaa507c297fe54b39b5. The PR diff no longer has the reflow or the trailing-comma changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057#issuecomment-6037381975
@1fanwang welcome to contributors gang!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057#discussion_r4168186896
##########
table/snapshot_producers.go:
##########
@@ -543,6 +567,12 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
}
}
+ // Java's merge manager writes zero-count manifests; this path omits bins
+ // with no live entries.
Review Comment:
Nit: "no live entries" isn't quite the condition. A bin holding only deletes from *this* snapshot is still written. A bin is dropped only when all of its entries are deletes from earlier snapshots:
```suggestion
// whose entries are all deletes from earlier snapshots.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057#discussion_r4168876003
##########
table/snapshot_producers_test.go:
##########
@@ -578,6 +578,208 @@ func TestManifestMergeManagerClosesWriterOnError(t *testing.T) {
require.ErrorIs(t, err, errLimitedWrite)
}
+func TestManifestMergeManagerClosesWriterBeforeFileOnReadFailure(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+ df := newTestDataFile(t, spec, "file://data-1.parquet", nil)
+ manifest := writeTestManifestWithContent(t, trackIO, spec, schema, sp.snapshotID,
+ "table-location/metadata/manifest-1.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusADDED, &sp.snapshotID, nil, nil, df),
+ })
+ missingManifest := iceberg.NewManifestFile(2, "table-location/metadata/missing.avro", 1,
+ int32(spec.ID()), sp.snapshotID).Build()
+
+ mgr := manifestMergeManager{snap: sp}
+ _, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifest, missingManifest})
+ require.ErrorIs(t, err, fs.ErrNotExist)
+ require.NotContains(t, err.Error(), "write after close",
+ "buffered entries must flush before the underlying file closes")
+ require.Equal(t, 1, trackIO.GetWriterCount())
+ require.Empty(t, trackIO.GetUnclosedWriters())
+}
+
+func TestManifestMergeReturnsReadErrorBeforeCreatingWriter(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+
+ oldSnapshotID := sp.snapshotID - 1
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ deletedManifest := writeTestManifestWithContent(t, trackIO, spec, schema, oldSnapshotID,
+ "table-location/metadata/historical-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &oldSnapshotID, &sequenceNumber, nil, df),
+ })
+ missingManifest := iceberg.NewManifestFile(2, "table-location/metadata/missing.avro", 1,
+ int32(spec.ID()), sp.snapshotID).Build()
+
+ mgr := manifestMergeManager{snap: sp}
+ _, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{deletedManifest, missingManifest})
+ require.ErrorIs(t, err, fs.ErrNotExist)
+ require.Zero(t, trackIO.GetWriterCount())
+}
+
+func TestManifestMergeSkipsHistoricalDeletedOnlyManifest(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+
+ oldSnapshotID := sp.snapshotID - 1
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ manifestFile := writeTestManifestWithContent(t, trackIO, spec, schema, oldSnapshotID,
+ "table-location/metadata/historical-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &oldSnapshotID, &sequenceNumber, nil, df),
+ })
+
+ trackIO.writers = make(map[string]*trackingWriteCloser)
Review Comment:
`writeTestManifestWithContent` persists through `fs.WriteFile`, never `Create`, so no `trackingWriteCloser` is ever registered during setup and this reset just clears an already-empty map. It reads like setup writes go through `Create` and need clearing, which future tests might cargo-cult into a spot where it masks a real writer leak. I'd drop the line; the `require.Zero(t, trackIO.GetWriterCount())` below already covers it (and it also mutates the field without `writersMu`).
##########
table/snapshot_producers.go:
##########
@@ -527,12 +542,21 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
switch {
case entry.Status() == iceberg.EntryStatusDELETED && entry.SnapshotID() == m.snap.snapshotID:
+ if err = ensureWriter(); err != nil {
Review Comment:
The dead write into the named `err` is gone now that `ensureWriter` returns its own `writerErr`, nice. The one thing left is that the loop-scoped `err` from `for entry, err := range` at line 538 shadows the named return, and these new `ensureWriter` assignments make that shadow easier to misread. Renaming it to `iterErr` (or a one-line note) would settle it. Not blocking.
##########
table/snapshot_producers_test.go:
##########
@@ -578,6 +578,208 @@ func TestManifestMergeManagerClosesWriterOnError(t *testing.T) {
require.ErrorIs(t, err, errLimitedWrite)
}
+func TestManifestMergeManagerClosesWriterBeforeFileOnReadFailure(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+ df := newTestDataFile(t, spec, "file://data-1.parquet", nil)
+ manifest := writeTestManifestWithContent(t, trackIO, spec, schema, sp.snapshotID,
+ "table-location/metadata/manifest-1.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusADDED, &sp.snapshotID, nil, nil, df),
+ })
+ missingManifest := iceberg.NewManifestFile(2, "table-location/metadata/missing.avro", 1,
+ int32(spec.ID()), sp.snapshotID).Build()
+
+ mgr := manifestMergeManager{snap: sp}
+ _, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifest, missingManifest})
+ require.ErrorIs(t, err, fs.ErrNotExist)
+ require.NotContains(t, err.Error(), "write after close",
+ "buffered entries must flush before the underlying file closes")
+ require.Equal(t, 1, trackIO.GetWriterCount())
+ require.Empty(t, trackIO.GetUnclosedWriters())
+}
+
+func TestManifestMergeReturnsReadErrorBeforeCreatingWriter(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+
+ oldSnapshotID := sp.snapshotID - 1
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ deletedManifest := writeTestManifestWithContent(t, trackIO, spec, schema, oldSnapshotID,
+ "table-location/metadata/historical-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &oldSnapshotID, &sequenceNumber, nil, df),
+ })
+ missingManifest := iceberg.NewManifestFile(2, "table-location/metadata/missing.avro", 1,
+ int32(spec.ID()), sp.snapshotID).Build()
+
+ mgr := manifestMergeManager{snap: sp}
+ _, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{deletedManifest, missingManifest})
+ require.ErrorIs(t, err, fs.ErrNotExist)
+ require.Zero(t, trackIO.GetWriterCount())
+}
+
+func TestManifestMergeSkipsHistoricalDeletedOnlyManifest(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+
+ oldSnapshotID := sp.snapshotID - 1
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ manifestFile := writeTestManifestWithContent(t, trackIO, spec, schema, oldSnapshotID,
+ "table-location/metadata/historical-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &oldSnapshotID, &sequenceNumber, nil, df),
+ })
+
+ trackIO.writers = make(map[string]*trackingWriteCloser)
+
+ mgr := manifestMergeManager{snap: sp}
+ created, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifestFile})
+ require.NoError(t, err)
+ require.Nil(t, created)
+ require.Zero(t, trackIO.GetWriterCount())
+}
+
+func TestManifestMergeKeepsCurrentSnapshotDeletedEntries(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ txn, wfs := createTestTransactionWithMemIO(t, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, wfs, nil, nil)
+
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ manifestFile := writeTestManifestWithContent(t, wfs, spec, schema, sp.snapshotID,
+ "mem://default/table-location/metadata/current-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &sp.snapshotID, &sequenceNumber, nil, df),
+ })
+
+ mgr := manifestMergeManager{snap: sp}
+ created, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifestFile})
+ require.NoError(t, err)
+ require.NotNil(t, created)
+
+ var entries []iceberg.ManifestEntry
+ for entry, err := range created.Entries(wfs, false) {
+ require.NoError(t, err)
+ entries = append(entries, entry)
+ }
+ require.Len(t, entries, 1)
+ require.Equal(t, iceberg.EntryStatusDELETED, entries[0].Status())
+ require.Equal(t, sp.snapshotID, entries[0].SnapshotID())
+ require.Equal(t, df.FilePath(), entries[0].DataFile().FilePath())
+}
+
+func TestManifestMergeGroupDropsEmptyMergedBin(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ txn, wfs := createTestTransactionWithMemIO(t, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, wfs, nil, nil)
+
+ oldSnapshotID := sp.snapshotID - 1
+ sequenceNumber := int64(1)
+ manifests := make([]iceberg.ManifestFile, 0, 2)
+ for i := range 2 {
+ df := newTestDataFile(t, spec, "file://deleted-"+strconv.Itoa(i)+".parquet", nil)
+ manifests = append(manifests, writeTestManifestWithContent(t, wfs, spec, schema, oldSnapshotID,
+ "mem://default/table-location/metadata/historical-delete-"+strconv.Itoa(i)+".avro",
+ iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &oldSnapshotID, &sequenceNumber, nil, df),
+ }))
+ }
+
+ mgr := manifestMergeManager{
+ targetSizeBytes: manifests[0].Length() + manifests[1].Length(),
+ minCountToMerge: 1,
+ mergeEnabled: true,
+ mergeConcurrency: 1,
+ snap: sp,
+ }
+
+ merged, err := mgr.mergeGroup(manifests[0], spec.ID(), manifests)
+ require.NoError(t, err)
+ require.Empty(t, merged)
+}
+
+func TestMergeAppendCommitSkipsHistoricalDeleteOnlyBin(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ txn, wfs := createTestTransactionWithMemIO(t, spec)
+ schema := simpleSchema()
+ parentSnapshotID := txn.meta.newSnapshotID()
+ sequenceNumber := int64(1)
+ historicalManifests := make([]iceberg.ManifestFile, 0, 2)
+ for i := range 2 {
+ dataFile := newTestDataFile(t, spec, "file://deleted-"+strconv.Itoa(i)+".parquet", nil)
+ historicalManifests = append(historicalManifests, writeTestManifestWithContent(t, wfs, spec, schema,
+ parentSnapshotID, "mem://default/table-location/metadata/historical-delete-bin-"+strconv.Itoa(i)+".avro",
+ iceberg.ManifestContentData, []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &parentSnapshotID, &sequenceNumber, nil, dataFile),
+ }))
+ }
+ manifestListPath := "mem://default/table-location/metadata/snap-historical-delete-bin.avro"
+ manifestList, err := wfs.Create(manifestListPath)
+ require.NoError(t, err)
+ require.NoError(t, iceberg.WriteManifestList(
+ 2, manifestList, parentSnapshotID, nil, &sequenceNumber, 0, historicalManifests))
+ require.NoError(t, manifestList.Close())
+ parentSnapshot := &Snapshot{
+ SnapshotID: parentSnapshotID,
+ SequenceNumber: sequenceNumber,
+ TimestampMs: time.Now().UnixMilli(),
+ ManifestList: manifestListPath,
+ Summary: &Summary{Operation: OpAppend, Properties: iceberg.Properties{}},
+ }
+ require.NoError(t, txn.meta.AddSnapshot(parentSnapshot))
+ require.NoError(t, txn.meta.SetSnapshotRef(MainBranch, parentSnapshotID, BranchRef))
+ meta, err := txn.meta.Build()
+ require.NoError(t, err)
+
+ mergeTxn := New(Identifier{"db", "tbl"}, meta, "metadata.json", func(context.Context) (iceio.IO, error) {
+ return wfs, nil
+ }, nil).NewTransaction()
+ if mergeTxn.meta.props == nil {
+ mergeTxn.meta.props = make(iceberg.Properties)
+ }
+ mergeTxn.meta.props[ManifestMergeEnabledKey] = "true"
+ mergeTxn.meta.props[ManifestMinMergeCountKey] = "2"
+ mergeTxn.meta.props[ManifestTargetSizeBytesKey] = strconv.FormatInt(
+ historicalManifests[0].Length()+historicalManifests[1].Length(), 10)
+ mergeAppend := newMergeAppendFilesProducer(OpAppend, mergeTxn, wfs, nil, nil)
+ newFile := newTestDataFile(t, spec, "file://new-data.parquet", nil)
+ mergeAppend.appendDataFile(newFile)
+ updates, requirements, err := mergeAppend.commit(context.Background())
+ require.NoError(t, err)
+ require.NoError(t, mergeTxn.apply(updates, requirements))
+ meta, err = mergeTxn.meta.Build()
+ require.NoError(t, err)
+
+ outputManifests, err := meta.CurrentSnapshot().Manifests(wfs)
+ require.NoError(t, err)
+ require.Len(t, outputManifests, 1)
+ require.Equal(t, int32(1), outputManifests[0].AddedDataFiles())
+ var outputEntries []iceberg.ManifestEntry
+ for entry, entryErr := range outputManifests[0].Entries(wfs, false) {
+ require.NoError(t, entryErr)
+ outputEntries = append(outputEntries, entry)
+ }
+ require.Len(t, outputEntries, 1)
+ require.Equal(t, iceberg.EntryStatusADDED, outputEntries[0].Status())
+ require.Equal(t, newFile.FilePath(), outputEntries[0].DataFile().FilePath())
+ t.Logf("snapshot commit succeeded with %d output manifest containing the new data file", len(outputManifests))
Review Comment:
`len(outputManifests)` is already pinned by `require.Len(..., 1)` just above, so this log is noise on the success path. I'd drop it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix(table): skip empty historical-delete manifest merges [iceberg-go]
Posted by "1fanwang (via GitHub)" <gi...@apache.org>.
1fanwang commented on code in PR #2057:
URL: https://github.com/apache/iceberg-go/pull/2057#discussion_r4162555275
##########
table/snapshot_producers.go:
##########
@@ -507,17 +507,31 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
return nil, err
}
- wr, path, counter, fileCloser, err := m.snap.newManifestWriter(spec)
- if err != nil {
- return nil, err
- }
- defer internal.CheckedClose(fileCloser, &err)
+ var wr *iceberg.ManifestWriter
+ var path string
+ var counter *internal.CountingWriter
+ var fileCloser io.Closer
writerClosed := false
defer func() {
- if !writerClosed {
+ if wr != nil && !writerClosed {
internal.CheckedClose(wr, &err)
}
}()
+ defer func() {
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/4e958128577cf5ba785051ae0e3b6b09f481b8ef. Both cleanups now run in one deferred closure that closes the writer before the file.
##########
table/snapshot_producers.go:
##########
@@ -507,17 +507,31 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
return nil, err
}
- wr, path, counter, fileCloser, err := m.snap.newManifestWriter(spec)
- if err != nil {
- return nil, err
- }
- defer internal.CheckedClose(fileCloser, &err)
+ var wr *iceberg.ManifestWriter
+ var path string
+ var counter *internal.CountingWriter
+ var fileCloser io.Closer
writerClosed := false
defer func() {
- if !writerClosed {
+ if wr != nil && !writerClosed {
internal.CheckedClose(wr, &err)
}
}()
+ defer func() {
+ if fileCloser != nil {
+ internal.CheckedClose(fileCloser, &err)
+ }
+ }()
+
+ ensureWriter := func() error {
+ if wr != nil {
+ return nil
+ }
+
+ wr, path, counter, fileCloser, err = m.snap.newManifestWriter(spec)
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/4e958128577cf5ba785051ae0e3b6b09f481b8ef. The closure now uses its own local error.
##########
table/snapshot_producers.go:
##########
@@ -543,6 +566,10 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
}
}
+ if wr == nil {
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/4e958128577cf5ba785051ae0e3b6b09f481b8ef. The comment and the PR description now note the divergence from Java.
##########
table/snapshot_producers.go:
##########
@@ -577,7 +604,9 @@ func (m *manifestMergeManager) mergeGroup(firstManifest iceberg.ManifestFile, sp
if err != nil {
return nil, err
}
- output = append(output, created)
+ if created != nil {
Review Comment:
Agreed that it's separate. I left singleton bins unchanged here because this fix only covers bins that are actually merged, so #2037 still needs a follow-up for the lone tombstone manifest.
##########
table/snapshot_producers_test.go:
##########
@@ -578,6 +578,93 @@ func TestManifestMergeManagerClosesWriterOnError(t *testing.T) {
require.ErrorIs(t, err, errLimitedWrite)
}
+func TestManifestMergeSkipsHistoricalDeletedOnlyManifest(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ trackIO := newTrackingIO()
+ txn := createTestTransaction(t, trackIO, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, trackIO, nil, nil)
+
+ oldSnapshotID := sp.snapshotID - 1
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ manifestFile := writeTestManifestWithContent(t, trackIO, spec, schema, oldSnapshotID,
+ "table-location/metadata/historical-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &oldSnapshotID, &sequenceNumber, nil, df),
+ })
+
+ trackIO.writers = make(map[string]*trackingWriteCloser)
+
+ mgr := manifestMergeManager{snap: sp}
+ created, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifestFile})
+ require.NoError(t, err)
+ require.Nil(t, created)
+ require.Zero(t, trackIO.GetWriterCount())
+}
+
+func TestManifestMergeKeepsCurrentSnapshotDeletedEntries(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+ txn, wfs := createTestTransactionWithMemIO(t, spec)
+ sp := newFastAppendFilesProducer(OpAppend, txn, wfs, nil, nil)
+
+ sequenceNumber := int64(1)
+ df := newTestDataFile(t, spec, "file://deleted.parquet", nil)
+ manifestFile := writeTestManifestWithContent(t, wfs, spec, schema, sp.snapshotID,
+ "mem://default/table-location/metadata/current-delete.avro", iceberg.ManifestContentData,
+ []iceberg.ManifestEntry{
+ iceberg.NewManifestEntry(iceberg.EntryStatusDELETED, &sp.snapshotID, &sequenceNumber, nil, df),
+ })
+
+ mgr := manifestMergeManager{snap: sp}
+ created, err := mgr.createManifest(spec.ID(), []iceberg.ManifestFile{manifestFile})
+ require.NoError(t, err)
+ require.NotNil(t, created)
+
+ var entries []iceberg.ManifestEntry
+ for entry, err := range created.Entries(wfs, false) {
+ require.NoError(t, err)
+ entries = append(entries, entry)
+ }
+ require.Len(t, entries, 1)
+ require.Equal(t, iceberg.EntryStatusDELETED, entries[0].Status())
+ require.Equal(t, sp.snapshotID, entries[0].SnapshotID())
+ require.Equal(t, df.FilePath(), entries[0].DataFile().FilePath())
+}
+
+func TestManifestMergeGroupDropsEmptyMergedBin(t *testing.T) {
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/4e958128577cf5ba785051ae0e3b6b09f481b8ef. The new test has a two-manifest bin whose second manifest fails to read, and it asserts that the read error comes back with no writer opened.
##########
table/snapshot_producers.go:
##########
@@ -1762,16 +1792,13 @@ func (sp *snapshotProducer) commitManifests(newManifests, addedContent []iceberg
// creates it).
baseHeadID := sp.txn.baseRefSnapshotID(branch)
- return []Update{
- addSnap,
- // Carry over the branch's existing retention settings so advancing
- // the ref on commit does not silently discard them. The update
- // encodes exactly the current ref's retention (settings the branch
- // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
- // applies a set-snapshot-ref as a pure replace, so this fully
- // determines the resulting ref rather than merging with the old one.
- sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef),
- }, []Requirement{
- AssertRefSnapshotID(branch, baseHeadID),
- }, nil
+ // Carry over the branch's existing retention settings so advancing
+ // the ref on commit does not silently discard them. The update
+ // encodes exactly the current ref's retention (settings the branch
+ // lacks stay 0 and are dropped by the `omitempty` tags); the catalog
+ // applies a set-snapshot-ref as a pure replace, so this fully
+ // determines the resulting ref rather than merging with the old one.
+ retainingSnapshotRef := sp.txn.meta.NewRetainingSnapshotRefUpdate(branch, sp.snapshotID, BranchRef)
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/dcc96bbbaa851bbfc2e66eaa507c297fe54b39b5. The commitManifests reflow and the reformats are gone, so the PR diff only touches snapshot_producers.go and its tests.
##########
table/snapshot_producers_test.go:
##########
@@ -578,6 +578,130 @@ func TestManifestMergeManagerClosesWriterOnError(t *testing.T) {
require.ErrorIs(t, err, errLimitedWrite)
}
+func TestManifestMergeManagerClosesWriterBeforeFileOnWriteFailure(t *testing.T) {
+ spec := iceberg.NewPartitionSpec()
+ schema := simpleSchema()
+
+ // Use a byte-limited IO that fails after the writer is opened and
+ // has written some data, but before all entries are processed.
+ mem := newMemIO(manifestHeaderSize(t, 2, spec, schema), errLimitedWrite)
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/4e958128577cf5ba785051ae0e3b6b09f481b8ef. The test now uses trackingIO and asserts that the error has no write after close, so reversing the close order fails it.
##########
table/snapshot_producers.go:
##########
@@ -507,18 +507,32 @@ func (m *manifestMergeManager) createManifest(specID int, bin []iceberg.Manifest
return nil, err
}
- wr, path, counter, fileCloser, err := m.snap.newManifestWriter(spec)
- if err != nil {
- return nil, err
- }
- defer internal.CheckedClose(fileCloser, &err)
+ var wr *iceberg.ManifestWriter
+ var path string
+ var counter *internal.CountingWriter
+ var fileCloser io.Closer
writerClosed := false
+ // Close the ManifestWriter before the underlying file so a mid-loop
+ // write failure flushes into an open file, not a closed one.
defer func() {
- if !writerClosed {
+ if wr != nil && !writerClosed {
internal.CheckedClose(wr, &err)
}
+ if fileCloser != nil {
+ internal.CheckedClose(fileCloser, &err)
+ }
}()
+ ensureWriter := func() error {
+ if wr != nil {
+ return nil
+ }
+
+ wr, path, counter, fileCloser, err = m.snap.newManifestWriter(spec)
Review Comment:
Done in https://github.com/apache/iceberg-go/commit/4e958128577cf5ba785051ae0e3b6b09f481b8ef.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org