You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "debugmiller (via GitHub)" <gi...@apache.org> on 2026/07/04 20:54:50 UTC
[I] Manifests with all-DELETED entries are silently retained during manifest list construction [iceberg-go]
debugmiller opened a new issue, #1393:
URL: https://github.com/apache/iceberg-go/issues/1393
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
`overwriteFiles.existingManifests` iterates each parent snapshot manifest with `discardDeleted=true`, which skips entries with `status=DELETED`. For a manifest where every entry is `DELETED`, the iterator yields nothing (since `foundDeletedCount=0` amd `len(notDeleted)=0`) which the code interprets as "manifest was unaffected by this operation" and retains instead of dropping it at https://github.com/apache/iceberg-go/blob/a7c491261d495dcb055033583db34695eabe81c2/table/snapshot_producers.go#L190-L194
I think the fix is small, at [L190](https://github.com/apache/iceberg-go/blob/a7c491261d49/table/snapshot_producers.go#L190), only retain the manifest if the iterator actually found live entries:
```diff
if foundDeletedCount == 0 {
- existingFiles = append(existingFiles, m)
+ if len(notDeleted) > 0 {
+ existingFiles = append(existingFiles, m)
+ }
+ // len(notDeleted)==0: all entries are DELETED and were skipped by
+ // discardDeleted=true. The manifest is empty — drop it.
continue
}
```
We observed this when using overwrite on small tables, which produces one DELETE manifest via `deletedFilesManifests`. Since these are never pruned, they accumulate: after N overwrites the current snapshot's manifest list contains N all-DELETED manifests alongside the live data manifests, causing a growing set of unnecessary snapshots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Manifests with all-DELETED entries are not dropped during manifest list construction [iceberg-go]
Posted by "debugmiller (via GitHub)" <gi...@apache.org>.
debugmiller commented on issue #1393:
URL: https://github.com/apache/iceberg-go/issues/1393#issuecomment-4950888326
Thank you. I have testing and indeed that commit contains the fix I need! Sorry I did not notice it before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Manifests with all-DELETED entries are not dropped during manifest list construction [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #1393:
URL: https://github.com/apache/iceberg-go/issues/1393#issuecomment-4895130954
@debugmiller Thanks for the detailed diagnosis, it's spot on for the commit you referenced (a7c4912, May 26), where `foundDeletedCount == 0` was checked before `len(notDeleted) == 0`, so an all-DELETED manifest got retained instead of dropped.
This was fixed on main by #1154 ("Avoid carrying empty overwrite manifests", a286d19, Jun 8), which reorders the two checks so the `len(notDeleted) == 0` case is caught first and the empty manifest is dropped — functionally the same as your proposed fix. Current main (table/snapshot_producers.go:191-198) already has this.
a7c4912 predates that fix, so a rebase onto current main should resolve the accumulation. Could you confirm you still see it on latest main?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Manifests with all-DELETED entries are not dropped during manifest list construction [iceberg-go]
Posted by "debugmiller (via GitHub)" <gi...@apache.org>.
debugmiller closed issue #1393: Manifests with all-DELETED entries are not dropped during manifest list construction
URL: https://github.com/apache/iceberg-go/issues/1393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org