You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ranflarion (via GitHub)" <gi...@apache.org> on 2026/10/23 15:55:53 UTC

[PR] fix: apply nested Parquet leaf pruning through FixedSizeList, ListView and Dictionary [datafusion]

ranflarion opened a new pull request, #26668:
URL: https://github.com/apache/datafusion/pull/26668

   ## Which issue does this PR close?
   
   - Closes #26667.
   
   ## Rationale for this change
   
   A table that declares a narrower element struct under a `FixedSizeList`, `ListView`, `LargeListView` or `Dictionary` column still reads every Parquet leaf. The read plan derives the emitted type with `type_for_leaf_subset`, which rejects partial selections under those wrappers, so the clip #24315 computes for them always falls back to a full read.
   
   ## What changes are included in this PR?
   
   `project_type_for_leaves` descends through the same wrappers as `clip_type`, rebuilding each around its projected child with the physical outer parameters (`FixedSizeList` keeps its size, `Dictionary` its key type). That is the type #24315's reader tests pin for those wrappers.
   
   ## What is the testing strategy for this PR?
   
   `type_for_leaf_subset_matches_clip_through_wrappers` checks that `type_for_leaf_subset` reproduces `clip_for_cast`'s type for every wrapper the clip descends through; on main it fails at `ListView`. `type_for_leaf_subset_rejects_unsupported_partial_wrapper` now uses `RunEndEncoded`, which stays opaque. A `FixedSizeList` section in `parquet_nested_schema_pruning.slt` pins `bytes_scanned` for the narrow table at 83 B against 174 B for the full schema; on main both read 174 B. `ListView` and `Dictionary` columns can't be declared in SQL, so the unit test is what covers them.
   
   ## Are there any user-facing changes?
   
   No API changes. Scans of such columns read fewer bytes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: apply nested Parquet leaf pruning through FixedSizeList, ListView and Dictionary [datafusion]

Posted by "kosiew (via GitHub) via github" <gi...@datafusion.apache.org>.
kosiew commented on code in PR #26668:
URL: https://github.com/apache/datafusion/pull/26668#discussion_r4549213421


##########
datafusion/datasource-parquet/src/nested_schema_pruning.rs:
##########
@@ -602,17 +624,48 @@ mod tests {
     /// must fall back instead of predicting a type the reader may not emit.
     #[test]
     fn type_for_leaf_subset_rejects_unsupported_partial_wrapper() {
-        let physical = DataType::FixedSizeList(
+        let physical = DataType::RunEndEncoded(
+            Arc::new(Field::new("run_ends", DataType::Int32, false)),
             Arc::new(Field::new(
-                "item",
+                "values",
                 struct_of(vec![int64("x"), utf8("pad")]),
                 true,
             )),
-            2,
         );
         assert!(type_for_leaf_subset(&physical, &[0]).is_none());
     }
 
+    /// The read plan keeps only [`clip_for_cast`]'s leaf offsets and derives
+    /// the emitted type from them, so [`type_for_leaf_subset`] must reproduce
+    /// the clipped type for every wrapper the clip descends through.
+    #[test]
+    fn type_for_leaf_subset_matches_clip_through_wrappers() {
+        let item = |fields| Arc::new(Field::new("item", struct_of(fields), true));
+        let wrappers: [fn(FieldRef) -> DataType; 6] = [
+            DataType::List,
+            DataType::LargeList,
+            DataType::ListView,
+            DataType::LargeListView,
+            |item| DataType::FixedSizeList(item, 2),
+            |item| {
+                DataType::Dictionary(
+                    Box::new(DataType::Int32),
+                    Box::new(item.data_type().clone()),
+                )
+            },
+        ];
+        for wrap in wrappers {
+            let physical = wrap(item(vec![int64("x"), utf8("pad")]));
+            let target = wrap(item(vec![int64("x")]));

Review Comment:
   Could we optionally extend this test with physical `Struct<skip: Wrapper<Struct<a,b>>, keep: Wrapper<Struct<pad,x>>>` and target `Struct<keep: Wrapper<Struct<x>>>`, asserting offsets `[3]` and an independently constructed expected type? This would cover cursor advancement past a dropped wrapper and selection of a non-leading child. Existing generic coverage is helpful, so this is not a blocker.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: apply nested Parquet leaf pruning through FixedSizeList, ListView and Dictionary [datafusion]

Posted by "ranflarion (via GitHub) via github" <gi...@datafusion.apache.org>.
ranflarion commented on PR #26668:
URL: https://github.com/apache/datafusion/pull/26668#issuecomment-6926534975

   @kosiew looks like we have the approvals, can we merge this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: apply nested Parquet leaf pruning through FixedSizeList, ListView and Dictionary [datafusion]

Posted by "alamb (via GitHub) via github" <gi...@datafusion.apache.org>.
alamb commented on PR #26668:
URL: https://github.com/apache/datafusion/pull/26668#issuecomment-6929024232

   Thanks @kosiew and @ranflarion 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: apply nested Parquet leaf pruning through FixedSizeList, ListView and Dictionary [datafusion]

Posted by "alamb (via GitHub) via github" <gi...@datafusion.apache.org>.
alamb merged PR #26668:
URL: https://github.com/apache/datafusion/pull/26668


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: apply nested Parquet leaf pruning through FixedSizeList, ListView and Dictionary [datafusion]

Posted by "ranflarion (via GitHub) via github" <gi...@datafusion.apache.org>.
ranflarion commented on code in PR #26668:
URL: https://github.com/apache/datafusion/pull/26668#discussion_r4551966679


##########
datafusion/datasource-parquet/src/nested_schema_pruning.rs:
##########
@@ -602,17 +624,48 @@ mod tests {
     /// must fall back instead of predicting a type the reader may not emit.
     #[test]
     fn type_for_leaf_subset_rejects_unsupported_partial_wrapper() {
-        let physical = DataType::FixedSizeList(
+        let physical = DataType::RunEndEncoded(
+            Arc::new(Field::new("run_ends", DataType::Int32, false)),
             Arc::new(Field::new(
-                "item",
+                "values",
                 struct_of(vec![int64("x"), utf8("pad")]),
                 true,
             )),
-            2,
         );
         assert!(type_for_leaf_subset(&physical, &[0]).is_none());
     }
 
+    /// The read plan keeps only [`clip_for_cast`]'s leaf offsets and derives
+    /// the emitted type from them, so [`type_for_leaf_subset`] must reproduce
+    /// the clipped type for every wrapper the clip descends through.
+    #[test]
+    fn type_for_leaf_subset_matches_clip_through_wrappers() {
+        let item = |fields| Arc::new(Field::new("item", struct_of(fields), true));
+        let wrappers: [fn(FieldRef) -> DataType; 6] = [
+            DataType::List,
+            DataType::LargeList,
+            DataType::ListView,
+            DataType::LargeListView,
+            |item| DataType::FixedSizeList(item, 2),
+            |item| {
+                DataType::Dictionary(
+                    Box::new(DataType::Int32),
+                    Box::new(item.data_type().clone()),
+                )
+            },
+        ];
+        for wrap in wrappers {
+            let physical = wrap(item(vec![int64("x"), utf8("pad")]));
+            let target = wrap(item(vec![int64("x")]));

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org