You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adragomir (via GitHub)" <gi...@apache.org> on 2024/07/31 11:41:22 UTC
[I] Support deep schema pruning and projection [datafusion]
adragomir opened a new issue, #11745:
URL: https://github.com/apache/datafusion/issues/11745
### Is your feature request related to a problem or challenge?
At the moment, Datafusion supports top-level column pruning - we have a mechanism, `projection: [usize]` where we detect, and pass through all the layers a set of top-level columns to get from a schema. The columns are inferred from the input and passed through all the layers (logical -> optimize -> physical). Some implementation can also take advantage of these to minimize the data read from storage at the lowest level .
However, for **deeply nested schemas** (a small number of huge deeply nested top-level column, list of structs with maps etc), this optimization is not so useful, because the actual top level columns are very large.
### Describe the solution you'd like
We should have a way to represent, and push through all the layers the "deep" projection of the actual leaves that we need in the query.
The schema and data returned after applying the deep schema pruning should reflect the changes (select 1 field from a struct in a list, we should get a list with a struct with a single field etc)
The feature needs to be applicable only to physical layouts that actually support it (for example Parquet and Arrow)
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3670337382
A blocker came up for pushing down field access expressions into scans: https://github.com/apache/datafusion/issues/19387
@AdamGS I would love to get your context / input from the Vortex side, I'm not familiar with what sorts of expressions you have / how this would apply to Vortex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-4143462909
Folks I think we are almost done here!
@friendlymatthew has pretty much finished this relay race with https://github.com/apache/datafusion/pull/20925
Datafusion now generally supports pushing down field access in structs into scans and can efficiently read single stuct fields from parquet.
The *one piece* that is missing is statistics support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3616942890
I think we’ll be in a good place to solve this by next week :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3633943256
Awesome! Review of #19111 is also welcome to keep this moving forward 😉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-2260801836
Possibly related to https://github.com/apache/datafusion/issues/2581
I think there may be some useful work that was added to arrow-rs https://github.com/apache/arrow-rs/issues/5148
cc @goldmedal who has been working on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-2260802331
Also, maybe @jayzhan211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3633914784
We do! I should have bandwidth next week 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-2266330803
I think it is worth to figure out the efficient way to deal with nested schema on arrow side. https://github.com/apache/datafusion/issues/2581. Then it will be clear how we could leverage on it. We might not need `scan_deep` hash map at the end 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3719385145
I'm currently working on https://github.com/apache/datafusion/issues/19550 which is an unforeseen blocker for this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3671194742
https://github.com/apache/datafusion/pull/19389 may be of interests to folks here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3671092889
I'm going to take tomorrow to read through it more thoroughly, but also pulling @gatesn here, I think we was having similar conversations internally recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-3633766628
https://github.com/apache/datafusion/pull/19111 is looking pretty close to getting merged.
I think the next step here would be for someone who actually works / needs struct column pushdown to pick this ticket up.
I did some prototyping in https://github.com/apache/datafusion/pull/18966/files?diff=unified#diff-bbd611d7b35d7f17633eebbf32a07dc9e394f20135754ed949751e8030049e38, maybe that's helpful to work off of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #11745: Support deep schema pruning and projection
URL: https://github.com/apache/datafusion/issues/11745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support deep schema pruning and projection [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11745:
URL: https://github.com/apache/datafusion/issues/11745#issuecomment-4150984332
Nice! Let's close this down and file a separate ticket to track statistics
Nice work @friendlymatthew
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org