You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/07/14 22:38:54 UTC
[I] Scan planning fails permanently with "Can't index multiple DVs" when a snapshot has duplicate deletion vectors for one data file (read-side recovery) [iceberg]
yadavay-amzn opened a new issue, #17206:
URL: https://github.com/apache/iceberg/issues/17206
### Apache Iceberg version
main (development)
### Query engine
Spark
### Please describe the bug 🐞
**Summary:** Once a table ends up with more than one deletion vector (DV) referencing the same data file in a snapshot, **every scan of that table fails during planning** with:
```
Can't index multiple DVs for <data-file>: <dv1> and <dv2>
```
thrown from `DeleteFileIndex.Builder.add(Map<String, DeleteFile> dvByPath, DeleteFile dv)` (`core/src/main/java/org/apache/iceberg/DeleteFileIndex.java`, ~L529-533). The table becomes effectively unreadable and there is no built-in way to recover it. This is core scan-planning code, so it affects every query engine, not just Spark.
**Why it throws:** The DV index is built by iterating all live delete-manifest entries in the snapshot (`DeleteFileIndex.Builder.build()` -> `loadDeleteFiles()` -> the loop at ~L501). For each DV, `add(...)` does `dvByPath.putIfAbsent(dv.referencedDataFile(), dv)` and throws a `ValidationException` if an entry already exists for that path. There is no deduplication or merge.
**Spec context:** The spec forbids more than one DV per data file per snapshot ("there can be at most one deletion vector for a given data file in a snapshot"; "Writers must ensure that there is at most one deletion vector per data file and must merge new deletes with existing vectors or position delete files"). So two DVs for one data file is always metadata corruption, and the `ValidationException` is a *correct* detection of an invalid state - but it leaves the table permanently unreadable.
**How a table reaches this state:** This is the read-side manifestation of a write-side conflict-detection gap (see #15966). When two concurrent row-level writers use a `conflictDetectionFilter` that is narrower than the set of data files they actually touch, `validateAddedDVs` can miss the conflict and allow two DVs for the same data file to be committed. #15966 fixes the *write* side and prevents future corruption, but it cannot heal tables that are **already** corrupted - those remain bricked.
**Steps to reproduce:** Produce a snapshot containing two live DV entries that reference the same data file (for example via the concurrent narrow-`conflictDetectionFilter` path described in #15966, or by crafting two delete manifests that each contain a DV for the same data file), then run any scan. Planning fails with `Can't index multiple DVs`.
**Expected behavior:** A reader should be able to recover from an already-corrupted table without data loss. Rather than throwing (and leaving the table unreadable), `DeleteFileIndex` should:
1. **Byte-identical duplicates** (same `location()`, `contentOffset()`, `contentSizeInBytes()`) - the same DV referenced from two manifest entries (e.g. a manifest-rewrite race): deduplicate silently, logging at DEBUG.
2. **Genuinely different DVs for the same data file**: **merge (union) their position bitmaps** by reading both Puffin blobs and combining them, analogous to the merge already performed on the write path, and log a WARNING about the spec violation. This preserves **all** deletes.
3. **Never silently pick one DV**, which would drop the other's deletes and resurrect deleted rows - a correctness violation worse than a scan failure.
**Proposed fix (I can contribute):** In `DeleteFileIndex.Builder.build()`, collect DVs per referenced data file, and in a post-pass deduplicate byte-identical DVs and union the position bitmaps of any genuinely different DVs (reusing the existing DV merge utility), logging a WARNING; index only the resulting merged DV. `FileIO` is available on the Builder. This is a read-side recovery/self-heal that complements the write-side fix in #15966. I would include a reproducer test.
### Willingness to contribute
- [X] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan planning fails permanently with "Can't index multiple DVs" when a snapshot has duplicate deletion vectors for one data file (read-side recovery) [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #17206:
URL: https://github.com/apache/iceberg/issues/17206#issuecomment-4982150028
I'm strongly opposed to changing this behavior. When we reach a condition which is in express violation of the spec, I don't think we should attempt to "fix" it on read. While we want to be generous in what input we handle, the moment that input violates known invariants I think we should break hard.
In the case that a table is broken, a user should be reverting back to a pre-corrupt state. Or manually repairing the table via some other means. I do not think we can just make assumptions about the state of the table (all dv's are valid and can be merged) and just successfully read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan planning fails permanently with "Can't index multiple DVs" when a snapshot has duplicate deletion vectors for one data file (read-side recovery) [iceberg]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on issue #17206:
URL: https://github.com/apache/iceberg/issues/17206#issuecomment-4996466854
Thanks for taking a look, Russell. You're right about the read path. Breaking hard on a spec violation is the correct default, and having the reader silently assume the DVs are all valid and safe to merge is an assumption I don't think we can stand behind either. I'll drop the read-time merge from #17207 so the hard fail at index build stays as it is.
I would like to get your thoughts on the following. Reverting to a pre-corrupt snapshot is the right first answer and covers most cases. The two spots where rollback alone seems to fall short are (a) the pre-corrupt snapshot may have already been expired, so there's nothing left to roll back to, and (b) rollback also discards any legitimate commits written after the corruption point, which can be a lot of data.
For just those cases, would an explicit, opt-in repair action be something you'd be open to? The idea would be a maintenance op the user runs knowingly, not anything on the read path, that writes a new spec-compliant snapshot with a single DV per data file. Union would be the resolution, and its main safety property is that it can only over-delete rather than resurrect a row that was already deleted, so it's the safer direction if we have to pick one automatically. That keeps the invariant intact everywhere by default and moves the "I accept this merge" decision to a human running maintenance rather than the reader.
If you think rollback plus manual repair already covers this well enough, I'm happy to close #17207 and leave the hard fail as the whole story. Mostly want to check whether the repair-action direction is worth pursuing in your mind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Scan planning fails permanently with "Can't index multiple DVs" when a snapshot has duplicate deletion vectors for one data file (read-side recovery) [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #17206:
URL: https://github.com/apache/iceberg/issues/17206#issuecomment-4996552609
A Repair Action would be fine, we just want it to be an explicit decision and not a silent automatic thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org