You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/30 06:06:37 UTC
[GitHub] [arrow-datafusion] jackwener opened a new issue, #4772: `PushDownProjection` doesn't push down the empty projection to the child plans of `Union`
jackwener opened a new issue, #4772:
URL: https://github.com/apache/arrow-datafusion/issues/4772
BTW, I'm not sure if it was a bug, but the `PushDownProjection` doesn't push down the empty projection to the child plans of `Union` in this example. Maybe @jackwener knows more about it.
_Originally posted by @HaoYang670 in https://github.com/apache/arrow-datafusion/pull/4753#discussion_r1058696036_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jefffrey commented on issue #4772: `PushDownProjection` doesn't push down the empty projection to the child plans of `Union`
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #4772:
URL: https://github.com/apache/arrow-datafusion/issues/4772#issuecomment-1401751195
Could you elaborate on the expected behaviour here? From the original discussion, regarding this test:
https://github.com/apache/arrow-datafusion/blob/5d4038a8463a575328bedbc22b32456f5dcd562c/datafusion/core/tests/sql/union.rs#L84-L98
I did an explain plan:
```
+------------------------------------------------------------+-----------------------------------------------------------------+
| plan_type | plan |
+------------------------------------------------------------+-----------------------------------------------------------------+
| initial_logical_plan | Projection: COUNT(UInt8(1)) |
| | Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1))]] |
| | Union |
| | Projection: t.a |
| | TableScan: t |
| | Projection: t.a |
| | TableScan: t |
| logical_plan after inline_table_scan | SAME TEXT AS ABOVE |
...
| logical_plan after common_sub_expression_eliminate | SAME TEXT AS ABOVE |
| logical_plan after push_down_projection | Projection: COUNT(UInt8(1)) |
| | Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1))]] |
| | Union |
| | TableScan: t projection=[a] |
| | TableScan: t projection=[a] |
| logical_plan after inline_table_scan | SAME TEXT AS ABOVE |
...
| logical_plan after push_down_projection | SAME TEXT AS ABOVE |
| logical_plan | Projection: COUNT(UInt8(1)) |
| | Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1))]] |
| | Union |
| | TableScan: t projection=[a] |
| | TableScan: t projection=[a] |
```
What would be the expected logical plan, if it's not this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `PushDownProjection` doesn't push down the empty projection to the child plans of `Union` [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #4772:
URL: https://github.com/apache/datafusion/issues/4772#issuecomment-3414733357
Closing per comments in #5745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] HaoYang670 commented on issue #4772: `PushDownProjection` doesn't push down the empty projection to the child plans of `Union`
Posted by "HaoYang670 (via GitHub)" <gi...@apache.org>.
HaoYang670 commented on issue #4772:
URL: https://github.com/apache/arrow-datafusion/issues/4772#issuecomment-1404486620
Hi @Jefffrey, actually I am not sure whether we should do this. It is just something appearing in my mind one day.
(sorry it should be the `aggregation functions` but not `empty projection` in the title)
For `union all`, we can push down the aggregation functions to the child plans. For example,
for a query `select count(*) from (select * from t1 union select * from t2)`, we can optimize the logical plan to count each child plan and then sum them together.
```
Projection: Sum(col1) as COUNT(UInt8(1))
Aggregate: groupBy=[[]], aggr=[[Sum(col1)]]
Union
Projection: COUNT(UInt8(1)) as col1
Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1))]]
TableScan: t projection=[a]
Projection: COUNT(UInt8(1)) as col1
Aggregate: groupBy=[[]], aggr=[[COUNT(UInt8(1))]]
TableScan: t projection=[a]
```
And it is similar for min, max, avg ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jefffrey commented on issue #4772: `PushDownProjection` doesn't push down the empty projection to the child plans of `Union`
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on issue #4772:
URL: https://github.com/apache/arrow-datafusion/issues/4772#issuecomment-1407096481
Ok I see what you mean now @HaoYang670
I suppose this could be a new logical optimizer rule `push_down_aggregate`, could be worth experimenting with I suppose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `PushDownProjection` doesn't push down the empty projection to the child plans of `Union` [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed issue #4772: `PushDownProjection` doesn't push down the empty projection to the child plans of `Union`
URL: https://github.com/apache/datafusion/issues/4772
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org