You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/09/07 23:42:47 UTC
[GitHub] [spark] bersprockets opened a new pull request, #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
bersprockets opened a new pull request, #37825:
URL: https://github.com/apache/spark/pull/37825
### What changes were proposed in this pull request?
In `RewriteDistinctAggregates`, when grouping aggregate expressions by function children, treat children that are semantically equivalent as the same.
### Why are the changes needed?
This PR will reduce the number of projections in the Expand operator when there are multiple distinct aggregations with superficially different children. In some cases, it will eliminate the need for an Expand operator.
Example: In the following query, the Expand operator creates 3\*n rows (where n is the number of incoming rows) because it has a projection for each of function children `b + 1`, `1 + b` and `c`.
```
create or replace temp view v1 as
select * from values
(1, 2, 3.0),
(1, 3, 4.0),
(2, 4, 2.5),
(2, 3, 1.0)
v1(a, b, c);
select
a,
count(distinct b + 1),
avg(distinct 1 + b) filter (where c > 0),
sum(c)
from
v1
group by a;
```
The Expand operator has three projections (each producing a row for each incoming row):
```
[a#87, null, null, 0, null, UnscaledValue(c#89)], <== projection #1 (for regular aggregation)
[a#87, (b#88 + 1), null, 1, null, null], <== projection #2 (for distinct aggregation of b + 1)
[a#87, null, (1 + b#88), 2, (c#89 > 0.0), null]], <== projection #3 (for distinct aggregation of 1 + b)
```
In reality, the Expand only needs one projection for `1 + b` and `b + 1`, because they are semantically equivalent.
With the proposed change, the Expand operator's projections look like this:
```
[a#67, null, 0, null, UnscaledValue(c#69)], <== projection #1 (for regular aggregations)
[a#67, (b#68 + 1), 1, (c#69 > 0.0), null]], <== projection #2 (for distinct aggregation on b + 1 and 1 + b)
```
With one less projection, Expand produces 2\*n rows instead of 3\*n rows, but still produces the correct result.
In the case where all distinct aggregates have semantically equivalent children, the Expand operator is not needed at all.
See benchmark results in the JIRA (SPARK-40382).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979537901
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -218,9 +218,16 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
val aggExpressions = collectAggregateExprs(a)
val distinctAggs = aggExpressions.filter(_.isDistinct)
+ val funcChildren = distinctAggs.flatMap { e =>
+ e.aggregateFunction.children.filter(!_.foldable)
+ }
+ val funcChildrenLookup = funcChildren.map { e =>
+ (e, funcChildren.find(fc => e.semanticEquals(fc)).getOrElse(e))
Review Comment:
```suggestion
(e, funcChildren.find(fc => e != fc && e.semanticEquals(fc)).getOrElse(e))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r980852866
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -254,7 +254,9 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
// Setup unique distinct aggregate children.
val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq.distinct
Review Comment:
not sure if this is necessary, but it's better to use `ExpressionSet(distinctAggGroups.keySet.flatten).toSeq`, instead of calling `.distinct` on `Seq[Expression]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #37825: [WIP][SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r990569446
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -254,7 +254,9 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
// Setup unique distinct aggregate children.
val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq.distinct
- val distinctAggChildAttrMap = distinctAggChildren.map(expressionAttributePair)
+ val distinctAggChildAttrMap = distinctAggChildren.map { e =>
+ e.canonicalized -> AttributeReference(e.sql, e.dataType, nullable = true)()
Review Comment:
It seems we can update `expressionAttributePair`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparksFyz commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
SparksFyz commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1241662233
https://github.com/apache/spark/pull/34953 take a look this pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979267219
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -213,7 +213,16 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
case a: Aggregate if mayNeedtoRewrite(a) => rewrite(a)
}
- def rewrite(a: Aggregate): Aggregate = {
+ def rewrite(aOrig: Aggregate): Aggregate = {
+ // Make children of distinct aggregations the same if they are only
+ // different due to superficial reasons, e.g.:
+ // "1 + col1" vs "col1 + 1", both should become "1 + col1"
+ // or
+ // "col1" vs "Col1", both should become "col1"
+ // This could potentially reduce the number of distinct
+ // aggregate groups, and therefore reduce the number of
+ // projections in Expand (or eliminate the need for Expand)
+ val a = reduceDistinctAggregateGroups(aOrig)
Review Comment:
Thanks! I am working on it, just working through some small complications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r980641159
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -218,9 +218,16 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
val aggExpressions = collectAggregateExprs(a)
val distinctAggs = aggExpressions.filter(_.isDistinct)
+ val funcChildren = distinctAggs.flatMap { e =>
+ e.aggregateFunction.children.filter(!_.foldable)
+ }
+ val funcChildrenLookup = funcChildren.map { e =>
+ (e, funcChildren.find(fc => e.semanticEquals(fc)).getOrElse(e))
+ }.toMap
+
// Extract distinct aggregate expressions.
val distinctAggGroups = aggExpressions.filter(_.isDistinct).groupBy { e =>
Review Comment:
Not sure if this is what you were hinting at, but for all maps related to distinct aggregation children, the code now uses `ExpressionSet` as a key. That way look-ups shouldn't care about superficial differences: the code never makes a lookup using an original child (...for the distinct aggregations. It still uses original children for regular aggregations).
>Then it's pretty easy to get back the original expressions, by ExpressionSet.toSeq.
By using `ExpressionSet` as the key to `distinctAggChildAttrLookup`, hopefully I don't need the originals at all.
Which is a good thing, since `ExpressionSet` is lossy when it comes to the originals, for example:
```
select count(distinct 1 + c1, c1 + 1), count(distinct c2 + 1, c2 + 2) from df;
```
This creates the following grouping keys for `distinctAggGroups`:
```
Set((1 + c1#106))
Set((c2#107 + 1), (c2#107 + 2))
```
`c1#106 + 1` is lost because of the way `ExpressionSet#add` works (it just ignores a new expression that is semantically equivalent to anything in `baseSet`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1275419730
> I still have this as WIP because I don't understand the impact, if any, of passing attributes created from the non-rewritten distinct aggregate expressions to the xxxAggregateExec constructor as `aggregateAttributes`
I removed the WIP designation from the title. I don't see any negative impact of passing attributes created from the non-rewritten distinct aggregate expressions to the xxxAggregateExec constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1243010034
> #34953 take a look this pr?
If I understand that PR correctly (and I may not), it is related but orthogonal. One PR doesn't preclude the other.
The PR you reference is making the `Expand` projections narrower (by reusing slots that have the same data type) and including the `then` expression of if-like expressions in the grouping of distinct aggregates -- thus potentially reducing the number of projections. I think that PR would still apply to the code even after my propose changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1257447790
It seems a little complex.
I have an idea to simplify the binary expressions in other optimizer rule.
Please reference `SimplifyBinaryComparison`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #37825: [WIP][SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1271731067
@cloud-fan I rebased and hopefully the tests will now run. Also I put [WIP] in the correct place in the title, which I will remove once I finished looking at the impact of using those attributes (as I mentioned above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1277732249
Thanks for all the help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
beliefer commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r976133494
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -413,6 +426,43 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
}}
}
+ private def reduceDistinctAggregateGroups(a: Aggregate): Aggregate = {
+ val aggExpressions = collectAggregateExprs(a)
+ val distinctAggs = aggExpressions.filter(_.isDistinct)
+
+ val funcChildren = distinctAggs.flatMap { e =>
+ e.aggregateFunction.children.filter(!_.foldable)
+ }
+
+ // For each function child, find the first instance that is semantically equivalent.
+ // E.g., assume funcChildren is the following three expressions:
+ // [('a + 1), (1 + 'a), 'b]
+ // then we want the map to be:
+ // Map(('a + 1) -> ('a + 1), (1 + 'a) -> ('a + 1), 'b -> 'b)
+ // That is, both ('a + 1) and (1 + 'a) map to ('a + 1).
+ // This is an n^2 operation, where n is the number of distinct aggregate children, but it
+ // happens only once every time this rule is called.
+ val funcChildrenLookup = funcChildren.map { e =>
+ (e, funcChildren.find(fc => e.semanticEquals(fc)).getOrElse(e))
Review Comment:
It seems the code lead to find out itself.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -413,6 +426,43 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
}}
}
+ private def reduceDistinctAggregateGroups(a: Aggregate): Aggregate = {
+ val aggExpressions = collectAggregateExprs(a)
+ val distinctAggs = aggExpressions.filter(_.isDistinct)
Review Comment:
These code duplicated, could you reuse one of them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
beliefer commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1258944783
> Thanks, I will take a look. This is reference to the fall-through case, where we discover there is really only a single distinct group, correct?
Yes. `distinct b + 1` and `distinct 1 + b` is the same one, and we could avoid apply `RewriteDistinctAggregates`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r980860055
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -402,7 +405,28 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
}
Aggregate(groupByAttrs, patchedAggExpressions, firstAggregate)
} else {
- a
+ // We may have one distinct group only because we grouped using ExpressionSet.
+ // To prevent SparkStrategies from complaining during sanity check, we need to check whether
+ // the original list of aggregate expressions had multiple distinct groups and, if so,
+ // patch that list so we have only one distinct group.
Review Comment:
I don't quite understand this. Shall we use `ExpressionSet` to fix issues in `SparkStrategies` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r985123478
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -402,7 +405,28 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
}
Aggregate(groupByAttrs, patchedAggExpressions, firstAggregate)
} else {
- a
+ // We may have one distinct group only because we grouped using ExpressionSet.
+ // To prevent SparkStrategies from complaining during sanity check, we need to check whether
+ // the original list of aggregate expressions had multiple distinct groups and, if so,
+ // patch that list so we have only one distinct group.
Review Comment:
>Shall we use ExpressionSet to fix issues in SparkStrategies as well?
Looking...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r967860756
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -1451,6 +1451,22 @@ class DataFrameAggregateSuite extends QueryTest
val df = Seq(1).toDF("id").groupBy(Stream($"id" + 1, $"id" + 2): _*).sum("id")
checkAnswer(df, Row(2, 3, 1))
}
+
+ test("SPARK-40382: All distinct aggregation children are semantically equivalent") {
Review Comment:
This test succeeds without the changes to RewriteDistinctAggregates. It's just a sanity test to check that the grouping by semantic equivalence doesn't break this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979489753
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -213,7 +213,16 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
case a: Aggregate if mayNeedtoRewrite(a) => rewrite(a)
}
- def rewrite(a: Aggregate): Aggregate = {
+ def rewrite(aOrig: Aggregate): Aggregate = {
+ // Make children of distinct aggregations the same if they are only
+ // different due to superficial reasons, e.g.:
+ // "1 + col1" vs "col1 + 1", both should become "1 + col1"
+ // or
+ // "col1" vs "Col1", both should become "col1"
+ // This could potentially reduce the number of distinct
+ // aggregate groups, and therefore reduce the number of
+ // projections in Expand (or eliminate the need for Expand)
+ val a = reduceDistinctAggregateGroups(aOrig)
Review Comment:
I made the change to use `ExpressionSet` and also commented on some of the issues.
I still prefer 'sanitizing' each original function child to use the first semantically equivalent child, in essence creating a new set of "original" children, as it bypasses some complexities (in particular the one where we may lose some of the original children as keys when we group by `ExpressionSet`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
URL: https://github.com/apache/spark/pull/37825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #37825: [SPARK-40382][SQL][WIP] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1271702293
> LGTM. Can you rebase to the latest master branch to retrigger github action jobs?
Will do.
I still have this as WIP because I don't understand the impact, if any, of passing attributes created from the non-rewritten distinct aggregate expressions to the xxxAggregateExec constructor as `aggregateAttributes` ([here](https://github.com/apache/spark/blob/e19b16f68c5114f97335273da57f833f051af6cc/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala#L263)). So I was going to review that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1252637037
cc @beliefer @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r976229259
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -213,7 +213,16 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
case a: Aggregate if mayNeedtoRewrite(a) => rewrite(a)
}
- def rewrite(a: Aggregate): Aggregate = {
+ def rewrite(aOrig: Aggregate): Aggregate = {
+ // Make children of distinct aggregations the same if they are only
+ // different due to superficial reasons, e.g.:
+ // "1 + col1" vs "col1 + 1", both should become "1 + col1"
+ // or
+ // "col1" vs "Col1", both should become "col1"
+ // This could potentially reduce the number of distinct
+ // aggregate groups, and therefore reduce the number of
+ // projections in Expand (or eliminate the need for Expand)
+ val a = reduceDistinctAggregateGroups(aOrig)
Review Comment:
shall we just canonicalize the function inputs when group by them? e.g. change `e.aggregateFunction.children.filter(!_.foldable).toSet` to `ExpressionSet(e.aggregateFunction.children.filter(!_.foldable))`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979487585
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -291,7 +298,8 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
val naf = if (af.children.forall(_.foldable)) {
af
} else {
- patchAggregateFunctionChildren(af) { x =>
+ patchAggregateFunctionChildren(af) { x1 =>
+ val x = funcChildrenLookup.getOrElse(x1, x1)
Review Comment:
Here's one of the complications, and my solution is somewhat brittle.
When grouping by `ExpressionSet`, in the case where there are superficially different children, we don't get all of the original children in the keys of `distinctAggGroups`. This is because multiple `ExpressionSet`s may have the same baseSet but different originals, and `groupBy` chooses only one `ExpressionSet` to represent the group's key (which is what want: we want `groupBy` to group by semantically equivalent children).
However, because `distinctAggGroups` is missing some original children in its keys, `distinctAggChildAttrLookup` is also missing some original children in its keys.
To bridge this gap, I used `funcChildrenLookup`. This data structure maps each original function child to the first semantically equivalent original function child. `funcChildrenLookup` will translate the original function child into the key (hopefully) expected by `distinctAggChildAttrLookup`. The brittleness is this: this code depends, at the very least, on which `ExpressionSet` is chosen by `groupBy` as the winner.
In the [first version](https://github.com/apache/spark/compare/master...bersprockets:spark:rewritedistinct_issue_orig?expand=1) of my PR, I modified the Aggregate (if needed) so there are no superfically different function children, thus there is no complexity when performing the groupings and the patching. I find it bit more straightforward to reason about.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -402,7 +410,21 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
}
Aggregate(groupByAttrs, patchedAggExpressions, firstAggregate)
} else {
- a
+ // We may have one distinct group only because we grouped using ExpressionSet.
+ // To prevent SparkStrategies from complaining during sanity check, we need to check whether
+ // the original list of aggregate expressions had multiple distinct groups and, if so,
+ // patch that list so we have only one distinct group.
+ if (funcChildrenLookup.keySet.size > funcChildrenLookup.values.toSet.size) {
+ val patchedAggExpressions = a.aggregateExpressions.map { e =>
Review Comment:
This is the second complexity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r979582134
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -218,9 +218,16 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
val aggExpressions = collectAggregateExprs(a)
val distinctAggs = aggExpressions.filter(_.isDistinct)
+ val funcChildren = distinctAggs.flatMap { e =>
+ e.aggregateFunction.children.filter(!_.foldable)
+ }
+ val funcChildrenLookup = funcChildren.map { e =>
+ (e, funcChildren.find(fc => e.semanticEquals(fc)).getOrElse(e))
+ }.toMap
+
// Extract distinct aggregate expressions.
val distinctAggGroups = aggExpressions.filter(_.isDistinct).groupBy { e =>
Review Comment:
Shall we return `ExpressionSet` in all branches so that the key type here is `ExpressionSet`? Then it's pretty easy to get back the original expressions, by `ExpressionSet.toSeq`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1258849538
@beliefer
> Please reference `SimplifyBinaryComparison`.
Thanks, I will take a look. This is reference to the fall-through case, where we discover there is really only a single distinct group, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] bersprockets commented on a diff in pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #37825:
URL: https://github.com/apache/spark/pull/37825#discussion_r992871693
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:
##########
@@ -254,7 +254,9 @@ object RewriteDistinctAggregates extends Rule[LogicalPlan] {
// Setup unique distinct aggregate children.
val distinctAggChildren = distinctAggGroups.keySet.flatten.toSeq.distinct
- val distinctAggChildAttrMap = distinctAggChildren.map(expressionAttributePair)
+ val distinctAggChildAttrMap = distinctAggChildren.map { e =>
+ e.canonicalized -> AttributeReference(e.sql, e.dataType, nullable = true)()
Review Comment:
`expressionAttributePair` is used in two other places, though, for regular aggregate children and filter expressions where the key does not need to be canonicalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #37825: [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in `RewriteDistinctAggregates`
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #37825:
URL: https://github.com/apache/spark/pull/37825#issuecomment-1277676830
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org