You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/05 17:16:56 UTC
[GitHub] [arrow-datafusion] andygrove opened a new issue, #2456: When creating `DFField` from `Expr` we should provide input plan not input schema
andygrove opened a new issue, #2456:
URL: https://github.com/apache/arrow-datafusion/issues/2456
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We often need to create a `DFField` to represent the output of an `Expr` in a schema. We typically attempt to do this today based on an input schema.
Examples:
- `fn to_field(&self, input_schema: &DFSchema)` in `ExprSchemable`
- `pub fn exprlist_to_fields<'a>(expr: impl IntoIterator<Item = &'a Expr>, plan: &LogicalPlan) -> Result<Vec<DFField>>`
This approach is problematic because the input schema loses a lot of information compared the plan it represents. For example, it might contain a column named `substr(c0, 1, 2)` and may not contain the column `c0`, making it impossible to reference `c0` later on. This use case comes up during aggregates and is at least partly the cause of bugs such as https://github.com/apache/arrow-datafusion/issues/2430
**Describe the solution you'd like**
We should change the signatures of the functions above to accept an input plan instead of an input schema. This gives us more control over resolving expressions.
**Describe alternatives you've considered**
None
**Additional context**
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove closed issue #2456: When creating `DFField` from `Expr` we should provide input plan not input schema
Posted by GitBox <gi...@apache.org>.
andygrove closed issue #2456: When creating `DFField` from `Expr` we should provide input plan not input schema
URL: https://github.com/apache/arrow-datafusion/issues/2456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on issue #2456: When creating `DFField` from `Expr` we should provide input plan not input schema
Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #2456:
URL: https://github.com/apache/arrow-datafusion/issues/2456#issuecomment-1235469470
This is no longer relevant thanks to https://github.com/apache/arrow-datafusion/pull/3336https://github.com/apache/arrow-datafusion/pull/3336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #2456: When creating `DFField` from `Expr` we should provide input plan not input schema
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #2456:
URL: https://github.com/apache/arrow-datafusion/issues/2456#issuecomment-1235405947
I think this may be closed with https://github.com/apache/arrow-datafusion/pull/3336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org