You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/16 07:39:45 UTC
[GitHub] [arrow-datafusion] mateuszkj opened a new issue, #2542: Window functions experssion should not be removed during optimalization step
mateuszkj opened a new issue, #2542:
URL: https://github.com/apache/arrow-datafusion/issues/2542
**Describe the bug**
`windowExpr` is removed fro Logical Plan during optimization. Which causes `Impossibly got empty window expression` error.
**To Reproduce**
```sql
RUST_LOG=debug datafusion-cli
WITH _sample_data AS (
SELECT 1 as a, 'aa' AS b
UNION ALL
SELECT 3 as a, 'aa' AS b
UNION ALL
SELECT 5 as a, 'bb' AS b
UNION ALL
SELECT 7 as a, 'bb' AS b
), _data2 AS (
SELECT
row_number() OVER (PARTITION BY s.b ORDER BY s.a) AS seq,
s.a,
s.b
FROM _sample_data s
)
SELECT d.b, MAX(d.a) AS max_a
FROM _data2 d
GROUP BY d.b
ORDER BY d.b;
-- Result
-- Error: Impossibly got empty window expression
```
**Expected behavior**
With above SQL i should got:
```sql
-- Result
-- aa; 3
-- bb; 7
```
**Additional context**
Input logical plan:
```
Sort: #d.b ASC NULLS LAST
Projection: #d.b, #MAX(d.a) AS max_a
Aggregate: groupBy=[[#d.b]], aggr=[[MAX(#d.a)]]
Projection: #_data2.seq, #_data2.a, #_data2.b, alias=d
Projection: #ROW_NUMBER() PARTITION BY [#s.b] ORDER BY [#s.a ASC NULLS LAST] AS seq, #s.a, #s.b, alias=_data2
WindowAggr: windowExpr=[[ROW_NUMBER() PARTITION BY [#s.b] ORDER BY [#s.a ASC NULLS LAST]]]
Projection: #_sample_data.a, #_sample_data.b, alias=s
Union
Projection: Int64(1) AS a, Utf8("aa") AS b
EmptyRelation
Projection: Int64(3) AS a, Utf8("aa") AS b
EmptyRelation
Projection: Int64(5) AS a, Utf8("bb") AS b
EmptyRelation
Projection: Int64(7) AS a, Utf8("bb") AS b
EmptyRelation
```
Optimized logical plan:
```
Sort: #d.b ASC NULLS LAST
Projection: #d.b, #MAX(d.a) AS max_a
Aggregate: groupBy=[[#d.b]], aggr=[[MAX(#d.a)]]
Projection: #_data2.a, #_data2.b, alias=d
Projection: #s.a, #s.b, alias=_data2
WindowAggr: windowExpr=[[]] <-------- Here is empty expression
Projection: #_sample_data.a, #_sample_data.b, alias=s
Union
Projection: Int64(1) AS a, Utf8("aa") AS b
EmptyRelation
Projection: Int64(3) AS a, Utf8("aa") AS b
EmptyRelation
Projection: Int64(5) AS a, Utf8("bb") AS b
EmptyRelation
Projection: Int64(7) AS a, Utf8("bb") AS b
EmptyRelation
```
We can see *windowExpr* is empty after optimization. I have tested this case on master ('[a825891](https://github.com/apache/arrow-datafusion/commit/a82589100d7b07cb629476a2df7eac002f48b8a0)').
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #2542: Unused Window functions experssion is wrongly removed from LogicalPlan during optimalization
Posted by GitBox <gi...@apache.org>.
alamb closed issue #2542: Unused Window functions experssion is wrongly removed from LogicalPlan during optimalization
URL: https://github.com/apache/arrow-datafusion/issues/2542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] AssHero commented on issue #2542: Unused Window functions experssion is wrongly removed from LogicalPlan during optimalization
Posted by GitBox <gi...@apache.org>.
AssHero commented on issue #2542:
URL: https://github.com/apache/arrow-datafusion/issues/2542#issuecomment-1140190324
I think this is because the window exprs are not referenced by any operators, and projection_push_down make empty new_window_expr for window logical plan.
If the window exprs are not referenced by any operators, we can remove the window exprs. cc @mateuszkj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org