You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "friendlymatthew (via GitHub)" <gi...@apache.org> on 2026/04/01 19:45:42 UTC
[I] [EPIC] first class support for struct field / Variant access in Parquet [datafusion]
friendlymatthew opened a new issue, #21308:
URL: https://github.com/apache/datafusion/issues/21308
This issue tracks the hard problems that block efficient querying of struct field columns in Parquet
To investigate, I used an MRE based on a variant column represented as a nested struct. Though, the findings generalize to any schema where queries access leaf fields inside struct columns
- https://github.com/apache/datafusion/issues/21306
- https://github.com/apache/datafusion/issues/21307
- https://github.com/apache/datafusion/pull/21003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] first class support for struct field / Variant access in Parquet [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21308:
URL: https://github.com/apache/datafusion/issues/21308#issuecomment-5219341392
> Splitting along "Variant as a first-class type" vs "make Parquet read less for struct columns" seems like the cleaner line to me, but you filed this one and have the fuller picture of where Variant is heading —
i agree this split makes sene to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] first class support for struct field / Variant access in Parquet [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #21308:
URL: https://github.com/apache/datafusion/issues/21308#issuecomment-5196307187
I've opened #24119 as an EPIC for the Parquet-side struct access work — projection pruning, filter pushdown, row-group pruning and the statistics that feed them — since a batch of concrete follow-ups just came out of #24090 (nested schema pruning for narrowed declared schemas) and they needed a home.
That leaves some overlap with this issue, so a suggestion: would it make sense to reframe this EPIC to focus specifically on **Variant**? The two linked issues here point in slightly different directions:
- #21306 (struct-aware optimizations are hardcoded to `GetFieldFunc`, so `variant_get` gets none of them) is squarely the Variant blocker, and it's the one that needs a general capability API rather than more struct-specific machinery. It's relevant to both EPICs; I've listed it under #24119 as shared rather than claiming it.
- #21307 (`ProjectionExec` produces unknown statistics for all `ScalarFunctionExpr` outputs) is really a physical-expr / statistics issue that happens to bite struct and Variant queries hardest — arguably it belongs to neither EPIC exclusively.
Splitting along "Variant as a first-class type" vs "make Parquet read less for struct columns" seems like the cleaner line to me, but you filed this one and have the fuller picture of where Variant is heading — happy to fold #24119 back in here instead if you'd rather keep it as one umbrella.
cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org