You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/07/09 16:41:03 UTC
[PR] refactor: share leaf-level parquet read plan assembly with the row filter [datafusion]
adriangb opened a new pull request, #23427:
URL: https://github.com/apache/datafusion/pull/23427
## Which issue does this PR close?
- N/A — follow-up cleanup requested in review of #23396 ([comment](https://github.com/apache/datafusion/pull/23396#discussion_r3552881821)).
**Stacked on #23396 and #23426.** Please merge those first; until they land this PR's diff against `main` also contains their commits. The change reviewable here is the last commit, `assemble_read_plan`.
## Rationale for this change
`projection_read_plan::build_projection_read_plan` and `row_filter::build_parquet_read_plan` both end in the same chain to build a leaf-level plan:
```
leaf_indices_for_roots → resolve_struct_field_leaves → extend/sort/dedup
→ ProjectionMask::leaves → build_filter_schema → ParquetReadPlan
```
They differ only in that the row filter also sizes the resulting leaves (`required_bytes`) and returns an `Option`. As @mbutrovich noted in review, a shared helper collapses both.
## What changes are included in this PR?
- New `assemble_read_plan(root_indices, struct_field_accesses, file_schema, schema_descr)` in `projection_read_plan.rs`. It returns the `ParquetReadPlan` plus the resolved leaf indices, which is what lets the row filter keep computing `required_bytes` without duplicating the leaf resolution.
- Both call sites delegate to it. `build_parquet_read_plan` shrinks from ~30 lines to ~10.
- `leaf_indices_for_roots`, `resolve_struct_field_leaves` and `build_filter_schema` are now private to `projection_read_plan` rather than `pub(crate)`, since `assemble_read_plan` is the only remaining caller.
No behavior change.
## Are these changes tested?
Yes, by existing tests: `datafusion-datasource-parquet` unit tests (158 pass) and the `parquet_integration` suite in `datafusion` (213 pass), both unchanged. The row-filter path is covered by `parquet::filter_pushdown` and the struct-field pushdown tests in `row_filter.rs`.
## Are there any user-facing changes?
No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BsRGsN18YBCji5iFTQmWpr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share leaf-level parquet read plan assembly with the row filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23427:
URL: https://github.com/apache/datafusion/pull/23427#issuecomment-4928749444
Folding this into #23426 — both cleanups came out of the same review thread on #23396, touch the same function, and are small enough to review as two commits in one PR. The `assemble_read_plan` commit is now the second commit there, unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] refactor: share leaf-level parquet read plan assembly with the row filter [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb closed pull request #23427: refactor: share leaf-level parquet read plan assembly with the row filter
URL: https://github.com/apache/datafusion/pull/23427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org