[I] Push down IS NULL / IS NOT NULL on struct columns into Parquet scan [datafusion]

["Druva-D (via GitHub)" <gi...@apache.org>]

Druva-D opened a new issue, #21795:
URL: https://github.com/apache/datafusion/issues/21795

   ### Is your feature request related to a problem or challenge?
   
   Filters like `WHERE struct_col IS NOT NULL` on struct columns cannot be pushed down into the Parquet scan. The `PushdownChecker` in `row_filter.rs` blanket-rejects any filter that references a whole struct column, even though `IS NOT NULL` only needs the struct's null bitmap — not any leaf data.
   
   This forces a `FilterExec` above the scan that materializes ALL leaf columns of the struct just to check nullability:
   
   ```
   FilterExec: price@4 IS NOT NULL
     DataSourceExec: projection=[..., price, ...]  -- reads all  price.* leaf columns
   ```
   
   For a struct with many fields, this means reading and decoding all leaf columns when semantically zero data is needed — only the struct's null bitmap.
   
   PRs #20822 and #20854 added pushdown support for `get_field` expressions on struct fields (e.g., `s['value'] > 10`), but `IS NULL` / `IS NOT NULL` on the whole struct was explicitly left unsupported. The test `struct_data_structures_prevent_pushdown` asserts this rejection.
   
   
   
   ### Describe the solution you'd like
   
   Allow `IS NULL` and `IS NOT NULL` on struct columns to be pushed down as row-level filters, reading only a single leaf column instead of all leaves.
   
   In Parquet, definition levels encode nullability at every nesting level independently. When arrow-rs reads even one leaf column, it reconstructs the struct's null bitmap from definition levels. `is_not_null()` then checks this bitmap, not the leaf's data.
   
   This means we can:
   1. Detect `IS NULL(Column(struct))` / `IS NOT NULL(Column(struct))` in `PushdownChecker::f_down` before the `Column` node is visited
   2. Project only the first leaf column of the struct in the `ProjectionMask`
   3. Let arrow-rs reconstruct the struct null bitmap from that single leaf's definition levels
   
   ### Describe alternatives you've considered
   
   1. **Definition-level-only reads**: Read only the definition levels of one leaf column without decoding the data values. This would be optimal (~1 bit per row vs 4+ bytes) but requires arrow-rs API changes (`ProjectionMask::definition_levels_only()` or similar) that don't exist today.
   
   2. **Statistics-based pruning only**: Use Parquet row-group null_count statistics to prune entire row groups. However, Parquet statistics are stored only for leaf columns, not struct columns, so struct-level null_count isn't directly available from metadata.
   
   
   
   ### Additional context
   
   Related to the filter pushdown EPIC: #20324
   Builds on: #20822 (struct field pushdown) and #20854 (leaf projection refinement)
   
   PR generated with [Claude Code](https://claude.com/product/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org