[I] feat(table): drop dangling equality delete files during RewriteDataFiles [iceberg-go]

["laskoviymishka (via GitHub)" <gi...@apache.org>]

laskoviymishka opened a new issue, #946:
URL: https://github.com/apache/iceberg-go/issues/946

   ### Feature Request / Improvement
   
   When `Transaction.RewriteDataFiles` compacts data files, position delete files are removed from the new snapshot if they pointed at rewritten files (good!). 
   
   Equality delete files are intentionally preserved (`collectSafePositionDeletes`, [`rewrite_data_files.go:201-235`](https://github.com/apache/iceberg-go/blob/main/table/rewrite_data_files.go#L201-L235)).
   
   For high-churn CDC workloads this leaves dangling equality delete entries in every subsequent snapshot's manifests forever. Compaction restores read perf in the short term (the seq-number filter at [`scanner.go:372-398`](https://github.com/apache/iceberg-go/blob/main/table/scanner.go#L372-L398) skips them), but the manifest read overhead grows linearly with the number of preserved eq-delete entries. Over days of replication this becomes the dominant scan cost.
   
   I'd like to propose extending the cleanup to also drop equality deletes that are provably dead after compaction.
   
   ### Why this matters
   
   Workload: PostgreSQL → Iceberg v2 CDC replication, ~one snapshot every 5 seconds, ~1 equality delete file per snapshot for UPDATE/DELETE operations.
   
   Empirical numbers from a small repro on a 10K-row table (Apple M1 Pro, local REST + MinIO):
   
   | State | Files (data / eq-delete) | `COUNT(*)` scan |
   |---|---|---|
   | Clean | 1 / 0 | ~30ms (baseline) |
   | 10K DML, no compaction | 16 / 120 | ~217ms (peak 382ms) |
   | **After `iceberg compact run`** | **1 / 120** | **~50ms** |
   
   So compaction is effective — 16 → 1 data file, scan time drops from 217ms to 50ms. But the 120 dangling eq-delete files now sit in every future snapshot's manifests with no purpose.
   
   Extrapolating sustained CDC at one eq-delete per snapshot:
   
   | Time running | Dangling eq-deletes | Estimated overhead |
   |---|---|---|
   | 1 hour | ~720 | ~120ms per scan |
   | 1 day | ~17,000 | ~3 sec per scan |
   | 1 week | ~120,000 | ~20 sec per scan |
   
   `ExpireSnapshots` doesn't help here, because the **current** snapshot's manifests still reference the dead eq-delete files. The cleanup has to happen during the rewrite commit itself.
   
   ### When is an equality delete safe to drop?
   
   The Iceberg v2 spec says an equality delete file E applies to a data file D iff `E.seq > D.seq` (and partitions match). After `RewriteDataFiles` rewrites a set of files in a partition, the new files have a sequence number strictly greater than every existing file in that partition.
   
   So an equality delete file E is **provably dead** after compaction when:
   
   1. Every data file with `data.seq < E.seq` in E's partition was in the rewrite set, AND
   2. `E.seq < new_compacted_file.seq` (always true — compaction commit gets the new highest seq)
   
   In other words: if compaction processed every data file E could have applied to, E will never apply to anything again.
   
   The simplest case — and the one most CDC sinks hit — is **unpartitioned full-partition compaction**: when every data file in the (only) partition is being rewritten, every equality delete file in that partition becomes dead.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] feat(table): drop dangling equality delete files during RewriteDataFiles [iceberg-go]

["zeroshade (via GitHub)" <gi...@apache.org>]

zeroshade closed issue #946: feat(table): drop dangling equality delete files during RewriteDataFiles
URL: https://github.com/apache/iceberg-go/issues/946


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org