You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2025/06/24 03:26:17 UTC
[I] [EPIC] expression pushdown and file level expression handling [datafusion]
adriangb opened a new issue, #16528:
URL: https://github.com/apache/datafusion/issues/16528
Several feature requests / issues have come up that I think can all be addressed with the groundwork being laid in https://github.com/apache/datafusion/pull/16461:
- https://github.com/apache/datafusion/pull/15261
- https://github.com/apache/datafusion/pull/15057
- https://github.com/apache/datafusion/issues/16004
- https://github.com/apache/datafusion/issues/14993
In particular, https://github.com/apache/datafusion/pull/16461 introduces a general framework for adapting an expression to a file's schema, handling any necessary casts and missing columns.
We can expand this by:
- Optimizing the expressions to minimize cost of casts, wip in https://github.com/pydantic/datafusion/pull/31. Closes https://github.com/apache/datafusion/issues/16004.
- Other optimizations passes, such as evaluating literals / nulls. Also related to https://github.com/apache/datafusion/issues/16004.
- Hook to handle missing columns (e.g. do something other than fill in with nulls based on Field metadata, could be a user defined default value); closes https://github.com/apache/datafusion/pull/15261
- Hook to transform an expression before or after it is rewritten for the physical file schema; closes https://github.com/apache/datafusion/pull/15057.
- Optimization to eliminate casts altogether when two types share the same parquet physical type (change the schema the data is read with and remove the cast).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] expression pushdown and file level expression handling [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #16528:
URL: https://github.com/apache/datafusion/issues/16528#issuecomment-3000748063
I added an example of how https://github.com/apache/datafusion/pull/16461 will help solve https://github.com/apache/datafusion/issues/14993 in https://github.com/apache/datafusion/commit/32725dd621ec5e96caf1970433f3549dca977a80. The hard bit will be piping the projection expressions around (adding methods to ExecutionPlan, deprecating (?) the passing around of Vec<usize> as the projection, etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org