You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jiacai2050 (via GitHub)" <gi...@apache.org> on 2024/02/02 09:35:19 UTC
[I] Wrong projection after optimize_projections rule [arrow-datafusion]
jiacai2050 opened a new issue, #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109
### Describe the bug
When upgrade latest datafusion from a old version, i found `optimize_projections` rule produce wrong projection.
```
create table t(x bigint, y bigint) as values (1,1), (2,2);
select x from t where y > 0;
```
Only x will be included in `projection`, y is missing from the plan.
### To Reproduce
First apply following changes, the purpose is to push down all filters
```diff
@@ -884,6 +884,7 @@ impl OptimizerRule for PushDownFilter {
let results = scan
.source
.supports_filters_pushdown(filter_predicates.as_slice())?;
+ let results = vec![TableProviderFilterPushDown::Exact; results.len()];
let zip = filter_predicates.iter().zip(results);
```
Then run datafusion-cli:
```
create table t(x bigint, y bigint) as values (1,1), (2,2);
explain verbose select x from t where y > 0;
```
Then we will get
```
| logical_plan after optimize_projections | Projection: t.x |
| | TableScan: t projection=[x], unsupported_filters=[t.y > Int64(0)] |
```
### Expected behavior
The right projection should be `[x, y]`.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 commented on issue #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109#issuecomment-1953408237
@mustafasrepo Sorry for my delayed response, I will re-check this using latest main branch this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 closed issue #9109: Wrong projection after optimize_projections rule
URL: https://github.com/apache/arrow-datafusion/issues/9109
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109#issuecomment-1929129812
I am confused why remove `column` in `filter` from `projection`? And the fix pr #9131 seems just adding it back?
https://github.com/apache/arrow-datafusion/pull/7670
@alamb @mustafasrepo @Dandandan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on issue #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109#issuecomment-1923437480
We will take a look and address next week. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109#issuecomment-1924675347
Possibly related https://github.com/apache/arrow-datafusion/issues/9111
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109#issuecomment-1929417274
I think this is working as designed as explained in https://github.com/apache/arrow-datafusion/pull/9131#pullrequestreview-1865020767 , though perhaps we could improve the design.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "mustafasrepo (via GitHub)" <gi...@apache.org>.
mustafasrepo commented on issue #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109#issuecomment-1929506409
When I run the queries as you described in the issue body. I got the following plan:
```
logical_plan
TableScan: t projection=[x], unsupported_filters=[t.y > Int64(0)]
physical_plan
MemoryExec: partitions=1, partition_sizes=[1]
```
Considering [#9131 review](https://github.com/apache/arrow-datafusion/pull/9131#pullrequestreview-1865020767). I don't think these plans are wrong or sub-optimal. Can you try your queries in the main branch (not in the latest release) if possible. Because, I couldn't reproduce the logical plan in the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Wrong projection after optimize_projections rule [arrow-datafusion]
Posted by "jiacai2050 (via GitHub)" <gi...@apache.org>.
jiacai2050 commented on issue #9109:
URL: https://github.com/apache/arrow-datafusion/issues/9109#issuecomment-1966590729
Closed since this is expected, thanks everyone involved.
For other developers, if you have met this issue, you can check how I "workaround" this issue here
- https://github.com/apache/incubator-horaedb/blob/8d53620b06eeac16ca4197e9e7f484ce44a6fc6c/src/table_engine/src/provider.rs#L237-L255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org