You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "ewoodbury (via GitHub)" <gi...@apache.org> on 2026/10/13 18:47:23 UTC

[PR] fix(arrow): keep struct presence when its requested children are missing [iceberg-rust]

ewoodbury opened a new pull request, #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438

   ## Which issue does this PR close?
   
   - Closes #3378.
   
   ## What changes are included in this PR?
   
   If every field inside a struct was dropped and re-added after a Parquet data file was written, reading that file returned NULL for the whole struct on every row, as long as another column was also selected. The reader picks which Parquet columns to read by matching field ids. None of the struct's current fields exist in the file, so no column under the struct was read, and the reader lost track of which rows have the struct present. Selecting only the struct hid the bug, because the reader then fell back to reading every column.
   
   This PR reads one existing column under such a struct to recover which rows have it present, then drops that column. Iceberg Java does the same in [`PresenceColumnSelector`](https://github.com/apache/iceberg/blob/db43b88a6c0d470bd4f8acf08a18328750aeed20/parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java#L160-L240) (apache/iceberg#17320), and the results now match Java. The column is chosen by the same rules:
   
   - An inner struct picks first, so one column can cover the structs around it.
   - The column with the lowest repetition level wins, then the first one in the file.
   - Only columns with field ids count.
   - A column under a map brings in the whole map, because parquet-rs drops a map unless both its key and value are read. Java's `PruneColumns.map` also keeps the whole map once its key is selected.
   
   Java reads no column for a struct that can never be null. This PR still reads one, because without it the struct is missing from the batch and the read fails with `Missing required field`. The result is the same.
   
   The extra column must also pass the reader's existing type check: every field on its path needs a field id, and the columns it reads must convert to Iceberg types. Only files from non-Iceberg writers can fail that check. If no column under a struct passes, that struct reads as it does on `main`: NULL, or for a map value struct, the existing `Map field must have exactly 2 fields` error.
   
   A struct whose only selected field is in the file but fails the reader's type-promotion check also still reads as NULL, as on `main`. That check drops any column it rejects, so it is separate from this fix.
   
   Behavior changes:
   
   - A struct whose fields were all re-added now reads correctly. Before, it came back NULL on every row.
   - A required struct like that failed on `main` with `Missing required field`. It now reads.
   - A `map<k, struct>` whose value struct had every field re-added failed on `main` with `Map field must have exactly 2 fields`. It now reads.
   - When every old field of a struct is gone and another column is selected, a new field with an `initial-default` now fails with `FeatureUnsupported` (#3261) instead of reading the whole struct as NULL. On `main`, other reads of these structs already fail this way: with an old field kept, or with only the struct selected. Applying nested defaults stays in #3261.
   
   ## Are these changes tested?
   
   Yes.
   
   - `test_struct_presence_when_requested_children_missing` reads, through `ArrowReader`, a file where every struct had all its fields re-added under new ids. It follows the no-default cases of Java's `ReadFormatModelTests` from apache/iceberg#17320 and includes null and empty lists and maps. I read each column next to `id` on `main` and on this branch:
   
     | Column | `main` | This branch |
     | --- | --- | --- |
     | `s`: struct with a nested struct | NULL in all 4 rows | correct |
     | `t`: struct holding only a list | NULL in all 4 rows | correct |
     | `u`: struct holding only a map | NULL in all 4 rows | correct |
     | `l`: list of structs | NULL in all 4 rows | correct |
     | `m`: map with a struct value | `Map field must have exactly 2 fields` | correct |
     | `r`: required struct | `Missing required field: r` | correct |
   
   - `test_projection_mask_reads_one_presence_leaf` checks which column is read. It skips a list element, a struct with no field id, and a `uint64` column, then picks the `int` next to that `uint64` inside a nested struct, ahead of a later column at the same level. It also checks that selecting only the struct reads just that one column (on `main` it read the whole file), and that a struct whose only column is a map with a `uint64` value reads nothing extra.
   
   To check that the tests cover each part of the fix, I disabled the parts one at a time and confirmed that each change made a test fail.
   
   These pass locally:
   
   - `cargo test -p iceberg --lib --all-features`
   - `cargo clippy -p iceberg --all-targets --all-features -- -D warnings`
   - `cargo fmt --all -- --check`
   - `dev/check_license_notice.sh`
   
   ## AI Disclosure
   
   Drafted with AI assistance (Grok and Claude). I fully own the fix strategy and its correctness.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(arrow): keep struct presence when its requested children are missing [iceberg-rust]

Posted by "ewoodbury (via GitHub) via issues" <is...@iceberg.apache.org>.
ewoodbury commented on PR #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438#issuecomment-6763723394

   @comphead / @laskoviymishka gentle ping. This is  follow-up from #3255 on struct-presence, and confirmed this fixes behavior to match iceberg-java. Thx!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(arrow): keep struct presence when its requested children are missing [iceberg-rust]

Posted by "ewoodbury (via GitHub) via issues" <is...@iceberg.apache.org>.
ewoodbury commented on code in PR #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438#discussion_r4544231435


##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -69,22 +70,30 @@ impl ArrowReader {
 
     /// Recursively extract leaf field IDs because Parquet projection works at the leaf column level.
     /// Nested types (struct/list/map) are flattened in Parquet's columnar format.
-    fn include_leaf_field_id(field: &NestedField, field_ids: &mut Vec<i32>) {
+    /// Also collects the ids of non-empty structs, which may need a presence leaf.
+    fn include_leaf_field_id(

Review Comment:
   Makes sense, done - it's now `ProjectedFields::include`, with leaf_ids kept as a `Vec`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(arrow): keep struct presence when its requested children are missing [iceberg-rust]

Posted by "ewoodbury (via GitHub)" <gi...@apache.org>.
ewoodbury commented on PR #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438#issuecomment-6257439466

   Thanks @JeonDaehong much appreciated! 
   
   @comphead / @laskoviymishka if you have a chance to take a look, at this one correctness followup from your reviews over in #3255.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(arrow): keep struct presence when its requested children are missing [iceberg-rust]

Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka merged PR #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(arrow): keep struct presence when its requested children are missing [iceberg-rust]

Posted by "laskoviymishka (via GitHub) via issues" <is...@iceberg.apache.org>.
laskoviymishka commented on code in PR #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438#discussion_r4537029434


##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -395,6 +417,118 @@ fn parquet_field_id(field: &Field) -> Option<i32> {
         .and_then(|field_id| i32::from_str(field_id).ok())
 }
 
+struct PresenceCandidate {
+    repetition: u32,
+    /// Leaf indexes to read: one leaf, or a whole map.
+    leaves: Range<usize>,
+}
+
+/// One file leaf under each projected struct that no selected leaf covers, so the
+/// struct keeps its definition level. Like Java's `PresenceColumnSelector`, inner
+/// structs go first, and the leaf with the lowest repetition level wins, then the first.
+fn presence_leaves(
+    fields: &Fields,
+    projected_structs: &HashSet<i32>,
+    leaf_field_ids: &HashSet<i32>,
+) -> Vec<PresenceCandidate> {
+    if projected_structs.is_empty() {
+        return Vec::new();
+    }
+    let mut walk = PresenceWalk {
+        projected_structs,
+        leaf_field_ids,
+        index: 0,
+        chosen: Vec::new(),
+    };
+    for field in fields {
+        walk.visit(field, 0, false);
+    }
+    walk.chosen
+}
+
+struct PresenceWalk<'a> {
+    projected_structs: &'a HashSet<i32>,
+    leaf_field_ids: &'a HashSet<i32>,
+    index: usize,
+    chosen: Vec<PresenceCandidate>,
+}
+
+impl PresenceWalk<'_> {
+    /// Returns whether a selected leaf is under `field`, and if not, the best presence
+    /// candidate. A candidate also goes through the projection's type check, so what it
+    /// reads must convert and every field on its path needs a field id. Candidates are
+    /// only built under a projected struct, since nothing else can use them.
+    fn visit(
+        &mut self,
+        field: &Field,
+        repetition: u32,
+        in_projected: bool,
+    ) -> (bool, Option<PresenceCandidate>) {
+        // Unwrap once, like `Fields::filter_leaves`, to count leaves the same way.
+        let data_type = match field.data_type() {

Review Comment:
   Could we add a case to `test_projection_mask_reads_one_presence_leaf` with a Dictionary or FixedSizeList field under the struct? This counter mirrors arrow's [`filter_leaves`](https://github.com/apache/arrow-rs/blob/9ea30dfacb7003de9ffc87ef50231f18229fcdb5/arrow-schema/src/fields.rs#L147-L247) by hand, and its indexes feed both `filter_leaves` and `ProjectionMask::leaves`. Those arms are reachable through an `ARROW:schema` hint, but no test drives them through the walk, so a change in arrow's traversal would drift without a compile error.



##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -69,22 +70,30 @@ impl ArrowReader {
 
     /// Recursively extract leaf field IDs because Parquet projection works at the leaf column level.
     /// Nested types (struct/list/map) are flattened in Parquet's columnar format.
-    fn include_leaf_field_id(field: &NestedField, field_ids: &mut Vec<i32>) {
+    /// Also collects the ids of non-empty structs, which may need a presence leaf.
+    fn include_leaf_field_id(

Review Comment:
   Could the two accumulators travel as one value, e.g. `struct ProjectedFields { leaf_ids: Vec<i32>, struct_ids: HashSet<i32> }` with this walk as a method? The name would stop saying "leaf" while it also collects struct ids, and [`get_arrow_projection_mask_with_field_ids`](https://github.com/apache/iceberg-rust/blob/6bf5310569a88308da47f79e47bfe2faa2a20f06/crates/iceberg/src/arrow/reader/projection.rs#L159-L201) could take `&ProjectedFields` instead of a sixth positional param. `leaf_ids` should stay a `Vec`, since its order is used further down.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(arrow): keep struct presence when its requested children are missing [iceberg-rust]

Posted by "ewoodbury (via GitHub) via issues" <is...@iceberg.apache.org>.
ewoodbury commented on code in PR #3438:
URL: https://github.com/apache/iceberg-rust/pull/3438#discussion_r4544304133


##########
crates/iceberg/src/arrow/reader/projection.rs:
##########
@@ -395,6 +417,118 @@ fn parquet_field_id(field: &Field) -> Option<i32> {
         .and_then(|field_id| i32::from_str(field_id).ok())
 }
 
+struct PresenceCandidate {
+    repetition: u32,
+    /// Leaf indexes to read: one leaf, or a whole map.
+    leaves: Range<usize>,
+}
+
+/// One file leaf under each projected struct that no selected leaf covers, so the
+/// struct keeps its definition level. Like Java's `PresenceColumnSelector`, inner
+/// structs go first, and the leaf with the lowest repetition level wins, then the first.
+fn presence_leaves(
+    fields: &Fields,
+    projected_structs: &HashSet<i32>,
+    leaf_field_ids: &HashSet<i32>,
+) -> Vec<PresenceCandidate> {
+    if projected_structs.is_empty() {
+        return Vec::new();
+    }
+    let mut walk = PresenceWalk {
+        projected_structs,
+        leaf_field_ids,
+        index: 0,
+        chosen: Vec::new(),
+    };
+    for field in fields {
+        walk.visit(field, 0, false);
+    }
+    walk.chosen
+}
+
+struct PresenceWalk<'a> {
+    projected_structs: &'a HashSet<i32>,
+    leaf_field_ids: &'a HashSet<i32>,
+    index: usize,
+    chosen: Vec<PresenceCandidate>,
+}
+
+impl PresenceWalk<'_> {
+    /// Returns whether a selected leaf is under `field`, and if not, the best presence
+    /// candidate. A candidate also goes through the projection's type check, so what it
+    /// reads must convert and every field on its path needs a field id. Candidates are
+    /// only built under a projected struct, since nothing else can use them.
+    fn visit(
+        &mut self,
+        field: &Field,
+        repetition: u32,
+        in_projected: bool,
+    ) -> (bool, Option<PresenceCandidate>) {
+        // Unwrap once, like `Fields::filter_leaves`, to count leaves the same way.
+        let data_type = match field.data_type() {

Review Comment:
   Added, and this turned out to actually catch a new bug: since Arrow 60 bump ([merged yesterday](https://github.com/apache/iceberg-rust/pull/3257)), filter_leaves counts the columns inside a ListView, but the walk counted the whole ListView as one column so any column after it got a wrong index. 
   
   Updated so this walk now counts the same way Arrow does, and never picks a column inside a ListView, since Iceberg can't read those. The test covers that, including a struct nested inside a ListView.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org