You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/29 12:38:58 UTC
[GitHub] [spark] peter-toth opened a new pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
peter-toth opened a new pull request #32396:
URL: https://github.com/apache/spark/pull/32396
### What changes were proposed in this pull request?
This PR adds a new rule `PullOutGroupingExpressions` to pull out complex grouping expressions to a `Project` node under an `Aggregate`. These expressions are then referenced in both grouping expressions and aggregate expressions without aggregate functions to ensure that optimization rules don't change the aggregate expressions to invalid ones that no longer refer to any grouping expressions.
### Why are the changes needed?
If aggregate expressions (without aggregate functions) in an `Aggregate` node are complex then the `Optimizer` can optimize out grouping expressions from them and so making aggregate expressions invalid.
Here is a simple example:
```
SELECT not(t.id IS NULL) , count(*)
FROM t
GROUP BY t.id IS NULL
```
In this case the `BooleanSimplification` rule does this:
```
=== Applying Rule org.apache.spark.sql.catalyst.optimizer.BooleanSimplification ===
!Aggregate [isnull(id#222)], [NOT isnull(id#222) AS (NOT (id IS NULL))#226, count(1) AS c#224L] Aggregate [isnull(id#222)], [isnotnull(id#222) AS (NOT (id IS NULL))#226, count(1) AS c#224L]
+- Project [value#219 AS id#222] +- Project [value#219 AS id#222]
+- LocalRelation [value#219] +- LocalRelation [value#219]
```
where `NOT isnull(id#222)` is optimized to `isnotnull(id#222)` and so it no longer refers to any grouping expression.
Before this PR:
```
== Optimized Logical Plan ==
Aggregate [isnull(id#222)], [isnotnull(id#222) AS (NOT (id IS NULL))#234, count(1) AS c#232L]
+- Project [value#219 AS id#222]
+- LocalRelation [value#219]
```
and running the query throws an error:
```
Couldn't find id#222 in [isnull(id#222)#230,count(1)#226L]
java.lang.IllegalStateException: Couldn't find id#222 in [isnull(id#222)#230,count(1)#226L]
```
After this PR:
```
== Optimized Logical Plan ==
Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (id IS NULL))#230, count(1) AS c#228L]
+- Project [isnull(value#219) AS _groupingexpression#233]
+- LocalRelation [value#219]
```
and the query works.
### Does this PR introduce _any_ user-facing change?
Yes, the query works.
### How was this patch tested?
Added new UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829412029
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624452389
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
I think I'm leaving this PR as it is now.
But tested that https://github.com/peter-toth/spark/commit/ed374fe046f5bb0a65b2bf72129b37df9cdce2ea could work, but I need to cast `TreePatternBits` to `Expression`.
Although, I wonder if it would make sense to split plan and expression pruning in the future like this: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd and so this pruning (and probably there are similar ones) became simpler: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd#diff-57201016f79912c165715811d7f7f37e2acbef2ae7b241c3c8a0b928d0052eb5R61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829203917
**[Test build #138081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138081/testReport)** for PR 32396 at commit [`5a6367b`](https://github.com/apache/spark/commit/5a6367bb340055a11a1fe2c342523bc44236e69b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829456834
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138081/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-831753364
NOTE: rather, it seems this change's improved TPCDS performance, e.g., 250230 => 228723 in q23a (sf=20). Nice work, @peter-toth
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829433081
**[Test build #138081 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138081/testReport)** for PR 32396 at commit [`5a6367b`](https://github.com/apache/spark/commit/5a6367bb340055a11a1fe2c342523bc44236e69b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830576161
> LGTM if TPCDS result shows no perf regression
Running it, will post the results today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829350849
**[Test build #138085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138085/testReport)** for PR 32396 at commit [`ba2f0c7`](https://github.com/apache/spark/commit/ba2f0c7ed1268e19cc61d3da02d6ff516b26d9a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624452389
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
I think I'm leaving this PR as it is now.
But tested that https://github.com/peter-toth/spark/commit/ed374fe046f5bb0a65b2bf72129b37df9cdce2ea could work, just I need to cast `TreePatternBits` to `Expression`.
Although, I wonder if it would make sense to split plan and expression pruning in the future like this: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd and so this pruning (and probably there are similar use cases where we want to stop traversal) became simpler: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd#diff-57201016f79912c165715811d7f7f37e2acbef2ae7b241c3c8a0b928d0052eb5R61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624435242
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
Hmm, yes, this could work with some explicit casting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth edited a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth edited a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830576161
> LGTM if TPCDS result shows no perf regression
Running it, will post the results today.
I forgot to make `PullOutGroupingExpressions` non-excludable. Fixed it in https://github.com/apache/spark/pull/32396/commits/fcc400520960a2ab180ced05f288c4c8c12f6d8d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624452389
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
I think I'm leaving this PR as it is now.
But tested that https://github.com/peter-toth/spark/commit/ed374fe046f5bb0a65b2bf72129b37df9cdce2ea could work, just I need to cast `TreePatternBits` to `Expression`.
Although, I wonder if it would make sense to split plan and expression pruning in the future like this: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd and so this pruning (and probably there are other similar use cases where we want to stop traversal) became simpler: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd#diff-57201016f79912c165715811d7f7f37e2acbef2ae7b241c3c8a0b928d0052eb5R61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830439074
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42637/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624151655
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
Review comment:
It seem like you only removed it from the following `map` method, but this `filterNot` can also be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830580193
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42646/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830434882
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42637/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830582118
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42646/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829350849
**[Test build #138085 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138085/testReport)** for PR 32396 at commit [`ba2f0c7`](https://github.com/apache/spark/commit/ba2f0c7ed1268e19cc61d3da02d6ff516b26d9a1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829532743
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138085/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829532743
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138085/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829412029
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829530512
**[Test build #138085 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138085/testReport)** for PR 32396 at commit [`ba2f0c7`](https://github.com/apache/spark/commit/ba2f0c7ed1268e19cc61d3da02d6ff516b26d9a1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624536755
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
Thanks a lot for exploring this, Peter! I'll think more about such use cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r623023416
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
+ .map {
+ case e if AggregateExpression.isAggregate(e) => e
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
Review comment:
we can stop traversal earlier: `case _ if e.foldable => e`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829411928
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830626860
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830495179
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138116/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829230515
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42601/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830433155
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42637/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624452389
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
https://github.com/peter-toth/spark/commit/ed374fe046f5bb0a65b2bf72129b37df9cdce2ea could work, but I need to cast `TreePatternBits` to `Expression`.
Although, I wonder if it would make sense to split plan and expression pruning like this: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd and so this pruning became simpler: https://github.com/peter-toth/spark/commit/d817fc724a78b8aa2de59999f237346d412e46dd#diff-57201016f79912c165715811d7f7f37e2acbef2ae7b241c3c8a0b928d0052eb5R61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830626860
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138125/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829352341
> The code change looks good. Can we run a TPCDS benchmark to make sure there is no perf regression?
I will run it and post the results soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624448258
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
Anyway, it's just my small preference -- it seems neater to use framework functions if it works. Feel free to merge whatever you feel comfortable with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624447710
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
> But this would traverse on a.groupingExpressions too which is not needed.
You're right. I think the following would behave the same as the manual recursion:
a.aggregateExpressions.map(_.transformWithPruning(e => !(AggregateExpression.isAggregate(e) || e.fordable))({
// the first two original case branches can be skipped here.
.....
}).asInstanceOf....)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624430840
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
We need to do a manual tree traversal if we want to stop recursion earlier, e.g. `case _ if AggregateExpression.isAggregate(e) => e`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624429289
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
Can it be done with a.transformExpressions similar to this one:
https://github.com/databricks/runtime/blob/8aff141a8545c2ea1759230482928e25684acbe2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/PullOutNondeterministic.scala#L39-L41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829214944
The code change looks good. Can we run a TPCDS benchmark to make sure there is no perf regression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830491326
**[Test build #138116 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138116/testReport)** for PR 32396 at commit [`bfb85de`](https://github.com/apache/spark/commit/bfb85de5874a82ed757ad26a086ae7f42e142f72).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r623015906
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/complexTypesSuite.scala
##########
@@ -405,14 +407,6 @@ class ComplexTypesSuite extends PlanTest with ExpressionEvalHelper {
val arrayAggRel = relation.groupBy(
CreateArray(Seq('nullable_id)))(GetArrayItem(CreateArray(Seq('nullable_id)), 0))
checkRule(arrayAggRel, arrayAggRel)
-
Review comment:
This can be removed now. It is optimized to:
```
Aggregate [*id#0L], [CASE WHEN (0 = *id#0L) THEN (*id#0L + 1) END AS a#0L]
+- LocalRelation <empty>, [*id#0L, nullable_id#0L]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829456834
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138081/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r623176850
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
+ .map {
+ case e if AggregateExpression.isAggregate(e) => e
Review comment:
Fixed in https://github.com/apache/spark/pull/32396/commits/ba2f0c7ed1268e19cc61d3da02d6ff516b26d9a1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624433476
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
Does the following one work?
a.transformExpressionsWithPruning(e => !(AggregateExpression.isAggregate(e) || e.fordable)) {
....
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829203917
**[Test build #138081 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138081/testReport)** for PR 32396 at commit [`5a6367b`](https://github.com/apache/spark/commit/5a6367bb340055a11a1fe2c342523bc44236e69b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624212217
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
Review comment:
Should be ok after https://github.com/apache/spark/pull/32396/commits/bfb85de5874a82ed757ad26a086ae7f42e142f72
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830439074
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42637/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829243912
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42601/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624446331
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions.map {
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if e.foldable => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
+ complexGroupingExpressionMap.get(e.canonicalized).map(_.toAttribute).getOrElse(e)
+ case _ => e.mapChildren(replaceComplexGroupingExpressions)
+ }
+ }
+
+ val newAggregateExpressions = a.aggregateExpressions
+ .map(replaceComplexGroupingExpressions(_).asInstanceOf[NamedExpression])
Review comment:
But this would traverse on `a.groupingExpressions` too which is not needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r623020449
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
+ .map {
+ case e if AggregateExpression.isAggregate(e) => e
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r623176671
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
Review comment:
Ah, you are right. I run some experiments to make this rule part of the `Analyzer` as `PullOutNondeterministic` is, but realized that it I would require more changes and reverted. I forgot to remove this. Fixed in: https://github.com/apache/spark/pull/32396/commits/ba2f0c7ed1268e19cc61d3da02d6ff516b26d9a1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r623176981
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
+ .map {
+ case e if AggregateExpression.isAggregate(e) => e
+ case e if !e.foldable && e.children.nonEmpty =>
+ complexGroupingExpressionMap
+ .getOrElseUpdate(e.canonicalized, Alias(e, s"_groupingexpression")())
+ .toAttribute
+ case o => o
+ }
+ if (complexGroupingExpressionMap.nonEmpty) {
+ def replaceComplexGroupingExpressions(e: Expression): Expression = {
+ e match {
+ case _ if AggregateExpression.isAggregate(e) => e
+ case _ if complexGroupingExpressionMap.contains(e.canonicalized) =>
Review comment:
Thanks indeed, fixed in https://github.com/apache/spark/pull/32396/commits/ba2f0c7ed1268e19cc61d3da02d6ff516b26d9a1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830774705
Thanks all for the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r623020362
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
Review comment:
is this needed? IIUC the analyzer guarantees that grouping expression can't contain aggregate expressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830495179
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138116/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829205097
@sigmod, @cloud-fan this is the alternative PR to https://github.com/apache/spark/pull/31913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830417308
**[Test build #138116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138116/testReport)** for PR 32396 at commit [`bfb85de`](https://github.com/apache/spark/commit/bfb85de5874a82ed757ad26a086ae7f42e142f72).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830670216
TPCDS [benchmark](https://github.com/peter-toth/spark/commit/fb9de2e563d712227bd7178dca4ed5a66f06d400) on scaleFactor=5 data looks good, no significant change: [TPCDSQueryBenchmark-results.txt](https://github.com/peter-toth/spark/blob/SPARK-34581-keep-grouping-expressions-2-benchmark/sql/core/benchmarks/TPCDSQueryBenchmark-results.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830573256
**[Test build #138125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138125/testReport)** for PR 32396 at commit [`fcc4005`](https://github.com/apache/spark/commit/fcc400520960a2ab180ced05f288c4c8c12f6d8d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on a change in pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on a change in pull request #32396:
URL: https://github.com/apache/spark/pull/32396#discussion_r624159920
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullOutGroupingExpressions.scala
##########
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import scala.collection.mutable
+
+import org.apache.spark.sql.catalyst.expressions.{Alias, Expression, NamedExpression}
+import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * This rule ensures that [[Aggregate]] nodes doesn't contain complex grouping expressions in the
+ * optimization phase.
+ *
+ * Complex grouping expressions are pulled out to a [[Project]] node under [[Aggregate]] and are
+ * referenced in both grouping expressions and aggregate expressions without aggregate functions.
+ * These references ensure that optimization rules don't change the aggregate expressions to invalid
+ * ones that no longer refer to any grouping expressions and also simplify the expression
+ * transformations on the node (need to transform the expression only once).
+ *
+ * For example, in the following query Spark shouldn't optimize the aggregate expression
+ * `Not(IsNull(c))` to `IsNotNull(c)` as the grouping expression is `IsNull(c)`:
+ * SELECT not(c IS NULL)
+ * FROM t
+ * GROUP BY c IS NULL
+ * Instead, the aggregate expression references a `_groupingexpression` attribute:
+ * Aggregate [_groupingexpression#233], [NOT _groupingexpression#233 AS (NOT (c IS NULL))#230]
+ * +- Project [isnull(c#219) AS _groupingexpression#233]
+ * +- LocalRelation [c#219]
+ */
+object PullOutGroupingExpressions extends Rule[LogicalPlan] {
+ override def apply(plan: LogicalPlan): LogicalPlan = {
+ plan transform {
+ case a: Aggregate if a.resolved =>
+ val complexGroupingExpressionMap = mutable.LinkedHashMap.empty[Expression, NamedExpression]
+ val newGroupingExpressions = a.groupingExpressions
+ .filterNot(AggregateExpression.containsAggregate)
Review comment:
Indeed, thanks. I made some other mistakes too in that commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830755716
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829243912
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42601/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830573256
**[Test build #138125 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138125/testReport)** for PR 32396 at commit [`fcc4005`](https://github.com/apache/spark/commit/fcc400520960a2ab180ced05f288c4c8c12f6d8d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #32396:
URL: https://github.com/apache/spark/pull/32396
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830417308
**[Test build #138116 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138116/testReport)** for PR 32396 at commit [`bfb85de`](https://github.com/apache/spark/commit/bfb85de5874a82ed757ad26a086ae7f42e142f72).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-831759715
Thanks @maropu for the extended test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] peter-toth edited a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
peter-toth edited a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830576161
> LGTM if TPCDS result shows no perf regression
Running it, will post the results today.
I forgot to make `PullOutGroupingExpressions` non-exludable. Fixed it in https://github.com/apache/spark/pull/32396/commits/fcc400520960a2ab180ced05f288c4c8c12f6d8d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-829401974
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830582118
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42646/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32396: [SPARK-34581][SQL] Don't optimize out grouping expressions from aggregate expressions without aggregate function
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32396:
URL: https://github.com/apache/spark/pull/32396#issuecomment-830622310
**[Test build #138125 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138125/testReport)** for PR 32396 at commit [`fcc4005`](https://github.com/apache/spark/commit/fcc400520960a2ab180ced05f288c4c8c12f6d8d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org