You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2026/08/20 16:12:57 UTC

[PR] feat(scan): [3/N] read and apply V3 deletion vectors [iceberg-rust]

mbutrovich opened a new pull request, #3035:
URL: https://github.com/apache/iceberg-rust/pull/3035

   ## Which issue does this PR close?
   
   - Closes #2792.
   
   ## What changes are included in this PR?
   
   Draft PR, stacked on #2868. This is Task 3 of the deletion-vector read epic: reading and applying V3 deletion vectors during a scan.
   
   - `CachingDeleteFileLoader` reads a deletion vector's `deletion-vector-v1` blob directly from its Puffin file by byte range (`content_offset` / `content_size_in_bytes`), decrypting first via `EncryptedInputFile` when the entry carries key metadata.
   - After decoding, the blob's cardinality is checked against the manifest entry's `record_count`, matching the same check Iceberg-Java does in `BitmapPositionDeleteIndex.deserializeBitmap`. Mismatch is a `DataInvalid` error, not a silent wrong result.
   - `DeleteFileIndex` now indexes deletion vectors by `referenced_data_file` and applies them ahead of position delete files, since a DV supersedes any position deletes for the same data file per spec.
   - A few spec invariants are now enforced as errors instead of silently mishandled:
     - a `PositionDeletes` entry with `content_offset` set but no `referenced_data_file`
     - two deletion vectors referencing the same data file (at most one is allowed per data file per snapshot)
     - a deletion vector whose partition or sequence number disagrees with the data file it claims to reference
   - `FileScanTaskDeleteFile` gained a `record_count` field, populated from the manifest entry, used for the cardinality check above.
   - `DeleteVector::deserialize` (from #2866) is now actually wired in; removed its `#[allow(dead_code)]`.
   
   No data-file read path changes outside of deletion vectors. Existing position delete and equality delete handling is unchanged.
   
   ## Are these changes tested?
   
   Yes.
   
   - Unit tests in `delete_file_index.rs` for: DV supersedes partition-scoped and path-scoped position deletes, DV coexists with equality deletes, rejecting a DV missing `referenced_data_file`, rejecting duplicate DVs for one data file, rejecting a DV with a mismatched partition or stale sequence number.
   - Unit tests in `caching_delete_file_loader.rs` for: reading and applying a DV, reading an encrypted DV, rejecting a cardinality mismatch, and the coordinate/cardinality validation helpers directly.
   - End-to-end tests in `positional_deletes.rs` reading a real Parquet data file with a DV applied through `ArrowReader`, including the cardinality-mismatch failure path through the full read.
   
   ## AI Disclosure
   
   Developed with the help of Claude Code, but I understand and support these changes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(scan): [3/N] read and apply V3 deletion vectors [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #3035:
URL: https://github.com/apache/iceberg-rust/pull/3035#discussion_r3919930554


##########
crates/iceberg/src/delete_file_index.rs:
##########
@@ -146,31 +180,83 @@ impl PopulatedDeleteFileIndex {
     /// Creates a new populated delete file index from a list of delete file contexts, which
     /// allows for fast lookup when determining which delete files apply to a given data file.
     ///
-    /// 1. Position deletes that reference a single data file, either through the
+    /// 1. A V3 deletion vector (a `PositionDeletes` entry stored as `Puffin`) is indexed by the
+    ///    `referenced_data_file` field, which the spec requires for deletion vectors.
+    ///    Fails if two deletion vectors reference the same data file: the spec allows at most
+    ///    one deletion vector per data file per snapshot.
+    /// 2. Other position deletes that reference a single data file, either through the
     ///    `referenced_data_file` field or through equal `file_path` column bounds,
     ///    are indexed by that data file's path.
-    /// 2. All other position deletes are indexed by the partition extracted from
+    /// 3. All other position deletes are indexed by the partition extracted from
     ///    their manifest entry.
-    /// 3. Equality deletes stored with an unpartitioned spec are applied as global
+    /// 4. Equality deletes stored with an unpartitioned spec are applied as global
     ///    deletes, per the spec. All other equality deletes are indexed by partition.
-    fn new(files: Vec<DeleteFileContext>) -> PopulatedDeleteFileIndex {
+    fn new(files: Vec<DeleteFileContext>) -> Result<PopulatedDeleteFileIndex> {
         let mut eq_deletes_by_partition: HashMap<Struct, Vec<Arc<DeleteFileContext>>> =
             HashMap::default();
         let mut pos_deletes_by_partition: HashMap<Struct, Vec<Arc<DeleteFileContext>>> =
             HashMap::default();
         let mut pos_deletes_by_path: HashMap<String, Vec<Arc<DeleteFileContext>>> =
             HashMap::default();
+        let mut dvs_by_referenced_data_file: HashMap<String, Arc<DeleteFileContext>> =
+            HashMap::default();
 
         let mut global_equality_deletes: Vec<Arc<DeleteFileContext>> = vec![];
 
-        files.into_iter().for_each(|ctx| {
+        for ctx in files {
             let arc_ctx = Arc::new(ctx);
 
-            let partition = arc_ctx.manifest_entry.data_file().partition();
+            let data_file = arc_ctx.manifest_entry.data_file();
+            let partition = data_file.partition();
 
             match arc_ctx.manifest_entry.content_type() {
                 DataContentType::PositionDeletes => {
-                    if let Some(path) = referenced_data_file(arc_ctx.manifest_entry.data_file()) {
+                    // A deletion vector is a position delete stored as a Puffin blob. The file
+                    // format is what distinguishes it from a position delete parquet file.
+                    if data_file.file_format() == DataFileFormat::Puffin {
+                        // The spec requires referenced_data_file, content_offset and
+                        // content_size_in_bytes on a deletion vector, so a missing one is a
+                        // malformed manifest entry, not an ordinary position delete to fall back
+                        // on.
+                        let Some(path) = data_file.referenced_data_file() else {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "deletion vector {} is missing referenced_data_file",
+                                    arc_ctx.manifest_entry.file_path()
+                                ),
+                            ));
+                        };
+
+                        if data_file.content_offset().is_none()
+                            || data_file.content_size_in_bytes().is_none()
+                        {
+                            return Err(Error::new(
+                                ErrorKind::DataInvalid,
+                                format!(
+                                    "deletion vector {} is missing content_offset or content_size_in_bytes",
+                                    arc_ctx.manifest_entry.file_path()
+                                ),
+                            ));
+                        }

Review Comment:
   These validations should be moved to FileScanTaskDeleteFile::builder()::build()
   
   I think it would be good to include this in 0.11 and would be happy if we can address this in a follow up 



##########
crates/iceberg/src/delete_file_index.rs:
##########
@@ -46,7 +57,10 @@ struct PopulatedDeleteFileIndex {
     eq_deletes_by_partition: HashMap<Struct, Vec<Arc<DeleteFileContext>>>,
     pos_deletes_by_partition: HashMap<Struct, Vec<Arc<DeleteFileContext>>>,
     pos_deletes_by_path: HashMap<String, Vec<Arc<DeleteFileContext>>>,
-    // TODO: Deletion Vector support
+    // V3 deletion vectors, keyed by the data file they apply to (referenced_data_file). At most
+    // one exists per data file per snapshot, and when one applies it supersedes any position
+    // delete files for that data file, partition-scoped or path-scoped alike.
+    dvs_by_referenced_data_file: HashMap<String, Arc<DeleteFileContext>>,

Review Comment:
   nit: we can improve the naming consistency here by updating pos_deletes_by_path to pos_deletes_by_referenced_data_file



##########
crates/iceberg/src/arrow/caching_delete_file_loader.rs:
##########
@@ -299,6 +330,137 @@ impl CachingDeleteFileLoader {
         }
     }
 
+    /// Validates a deletion-vector task and returns what the read needs as typed values:
+    /// `(start, len, referenced data file path, expected cardinality)`.
+    ///
+    /// The spec requires `referenced_data_file`, `content_offset` and `content_size_in_bytes` on
+    /// a deletion vector, and a deletion vector is always built from a manifest entry, so it
+    /// always carries `record_count`. A missing one is a manifest-entry inconsistency rather
+    /// than an I/O failure.
+    ///
+    /// Equality and ordinary position deletes have no equivalent validation in this loader: a
+    /// malformed equality/position delete file fails loudly when the Parquet reader can't open
+    /// it. A deletion vector's coordinates instead drive a raw byte-range read with no format
+    /// to fail against, so a bad coordinate would otherwise decode silently into the wrong (or
+    /// no) deletes, per the same corrupted-blob concern Iceberg-Java validates in
+    /// `BitmapPositionDeleteIndex.deserializeBitmap`.
+    fn validate_deletion_vector_task(

Review Comment:
   We should validate this when building `FileScanTaskDeleteFile`. 
   ```
   isDv == FileScanTaskDeleteFile.content_type == PosDel && FileScanTaskDeleteFile.file_format == puffin
   fn build() {
    if (isDv) { validate_deletion_vector_task() }
   }
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(scan): [3/N] read and apply V3 deletion vectors [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY merged PR #3035:
URL: https://github.com/apache/iceberg-rust/pull/3035


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org