You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "peter-toth (via GitHub)" <gi...@apache.org> on 2025/08/04 20:40:41 UTC
[PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
peter-toth opened a new pull request, #51820:
URL: https://github.com/apache/spark/pull/51820
### What changes were proposed in this pull request?
This is an alternative PR to https://github.com/apache/spark/pull/51810 to fix a regresion introduced in Spark 3.2 with https://github.com/apache/spark/pull/32470.
This PR defers the resolution of not fully resolved `UnresolvedHaving` nodes from `ResolveGroupingAnalytics`:
```
=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveGroupingAnalytics ===
'Sort ['s DESC NULLS LAST], true 'Sort ['s DESC NULLS LAST], true
!+- 'UnresolvedHaving ('count('product) > 2) +- 'UnresolvedHaving ('count(tempresolvedcolumn(product#261, product, false)) > 2)
! +- 'Aggregate [cube(Vector(0), Vector(1), product#261, region#262)], [product#261, region#262, sum(amount#263) AS s#264L] +- Aggregate [product#269, region#270, spark_grouping_id#268L], [product#269, region#270, sum(amount#263) AS s#264L]
! +- SubqueryAlias t +- Expand [[product#261, region#262, amount#263, product#266, region#267, 0], [product#261, region#262, amount#263, product#266, null, 1], [product#261, region#262, amount#263, null, region#267, 2], [product#261, region#262, amount#263, null, null, 3]], [product#261, region#262, amount#263, product#269, region#270, spark_grouping_id#268L]
! +- LocalRelation [product#261, region#262, amount#263] +- Project [product#261, region#262, amount#263, product#261 AS product#266, region#262 AS region#267]
! +- SubqueryAlias t
! +- LocalRelation [product#261, region#262, amount#263]
```
to `ResolveAggregateFunctions` to add the correct aggregate expressions:
```
=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveAggregateFunctions ===
'Sort ['s DESC NULLS LAST], true 'Sort ['s DESC NULLS LAST], true
!+- 'UnresolvedHaving (count(tempresolvedcolumn(product#261, product, false)) > cast(2 as bigint)) +- Project [product#269, region#270, s#264L]
! +- Aggregate [product#269, region#270, spark_grouping_id#268L], [product#269, region#270, sum(amount#263) AS s#264L] +- Filter (count(product)#272L > cast(2 as bigint))
! +- Expand [[product#261, region#262, amount#263, product#266, region#267, 0], [product#261, region#262, amount#263, product#266, null, 1], [product#261, region#262, amount#263, null, region#267, 2], [product#261, region#262, amount#263, null, null, 3]], [product#261, region#262, amount#263, product#269, region#270, spark_grouping_id#268L] +- Aggregate [product#269, region#270, spark_grouping_id#268L], [product#269, region#270, sum(amount#263) AS s#264L, count(product#261) AS count(product)#272L]
! +- Project [product#261, region#262, amount#263, product#261 AS product#266, region#262 AS region#267] +- Expand [[product#261, region#262, amount#263, product#266, region#267, 0], [product#261, region#262, amount#263, product#266, null, 1], [product#261, region#262, amount#263, null, region#267, 2], [product#261, region#262, amount#263, null, null, 3]], [product#261, region#262, amount#263, product#269, region#270, spark_grouping_id#268L]
! +- SubqueryAlias t +- Project [product#261, region#262, amount#263, product#261 AS product#266, region#262 AS region#267]
! +- LocalRelation [product#261, region#262, amount#263] +- SubqueryAlias t
! +- LocalRelation [product#261, region#262, amount#263]
```
### Why are the changes needed?
Fix a correctness isue described in https://github.com/apache/spark/pull/51810.
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a correctness issue.
### How was this patch tested?
Added new UT from https://github.com/apache/spark/pull/51810.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth closed pull request #51820: [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses
URL: https://github.com/apache/spark/pull/51820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51820:
URL: https://github.com/apache/spark/pull/51820#discussion_r2254411146
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -792,27 +792,33 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
} else {
colResolved.havingCondition
}
- // Try resolving the condition of the filter as though it is in the aggregate clause
- val (extraAggExprs, Seq(resolvedHavingCond)) =
- ResolveAggregateFunctions.resolveExprsWithAggregate(Seq(cond), aggForResolving)
-
- // Push the aggregate expressions into the aggregate (if any).
- val newChild = constructAggregate(h, selectedGroupByExprs, groupByExprs,
- aggregate.aggregateExpressions ++ extraAggExprs, aggregate.child)
-
- // Since the output exprId will be changed in the constructed aggregate, here we build an
- // attrMap to resolve the condition again.
- val attrMap = AttributeMap((aggForResolving.output ++ extraAggExprs.map(_.toAttribute))
- .zip(newChild.output))
- val newCond = resolvedHavingCond.transform {
- case a: AttributeReference => attrMap.getOrElse(a, a)
- }
-
- if (extraAggExprs.isEmpty) {
- Filter(newCond, newChild)
+ // `cond` might contain unresolved aggregate functions so defer its resolution to
+ // `ResolveAggregateFunctions` rule if needed.
+ if (!cond.resolved) {
Review Comment:
it's not only to minimize diff, but the "early return" pattern is recommended in general, see https://medium.com/swlh/return-early-pattern-3d18a41bba8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #51820:
URL: https://github.com/apache/spark/pull/51820#discussion_r2253583798
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -792,27 +792,33 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
} else {
colResolved.havingCondition
}
- // Try resolving the condition of the filter as though it is in the aggregate clause
- val (extraAggExprs, Seq(resolvedHavingCond)) =
- ResolveAggregateFunctions.resolveExprsWithAggregate(Seq(cond), aggForResolving)
-
- // Push the aggregate expressions into the aggregate (if any).
- val newChild = constructAggregate(h, selectedGroupByExprs, groupByExprs,
- aggregate.aggregateExpressions ++ extraAggExprs, aggregate.child)
-
- // Since the output exprId will be changed in the constructed aggregate, here we build an
- // attrMap to resolve the condition again.
- val attrMap = AttributeMap((aggForResolving.output ++ extraAggExprs.map(_.toAttribute))
- .zip(newChild.output))
- val newCond = resolvedHavingCond.transform {
- case a: AttributeReference => attrMap.getOrElse(a, a)
- }
-
- if (extraAggExprs.isEmpty) {
- Filter(newCond, newChild)
+ // `cond` might contain unresolved aggregate functions so defer its resolution to
+ // `ResolveAggregateFunctions` rule if needed.
+ if (!cond.resolved) {
Review Comment:
Honestly I'm not a big fan of `return` in Scala, but I can change it if minimizing diff is preffered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #51820:
URL: https://github.com/apache/spark/pull/51820#discussion_r2252597936
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -5057,6 +5057,22 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
}
+
+ test("SPARK-53094: Fix cube-related data quality problem") {
Review Comment:
This is the original author's contribution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #51820:
URL: https://github.com/apache/spark/pull/51820#issuecomment-3152397528
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "violetnspct (via GitHub)" <gi...@apache.org>.
violetnspct commented on PR #51820:
URL: https://github.com/apache/spark/pull/51820#issuecomment-3152645853
Maybe add tests for the following edge case?
Partially resolved conditions. Critical for ensuring correct resolution order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #51820:
URL: https://github.com/apache/spark/pull/51820#issuecomment-3156200502
Thanks @dongjoon-hyun , @cloud-fan for the review!
Merged to `master` (4.1.0).
I will open a backport PRs to 4.0.x and 3.5.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #51820:
URL: https://github.com/apache/spark/pull/51820#discussion_r2252623484
##########
sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:
##########
@@ -5057,6 +5057,22 @@ class SQLQuerySuite extends QueryTest with SharedSparkSession with AdaptiveSpark
}
}
}
+
+ test("SPARK-53094: Fix cube-related data quality problem") {
Review Comment:
Yes, this testcase is from @harris233's https://github.com/apache/spark/pull/51810, I just cleaned it up a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #51820:
URL: https://github.com/apache/spark/pull/51820#discussion_r2253554325
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -792,27 +792,33 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
} else {
colResolved.havingCondition
}
- // Try resolving the condition of the filter as though it is in the aggregate clause
- val (extraAggExprs, Seq(resolvedHavingCond)) =
- ResolveAggregateFunctions.resolveExprsWithAggregate(Seq(cond), aggForResolving)
-
- // Push the aggregate expressions into the aggregate (if any).
- val newChild = constructAggregate(h, selectedGroupByExprs, groupByExprs,
- aggregate.aggregateExpressions ++ extraAggExprs, aggregate.child)
-
- // Since the output exprId will be changed in the constructed aggregate, here we build an
- // attrMap to resolve the condition again.
- val attrMap = AttributeMap((aggForResolving.output ++ extraAggExprs.map(_.toAttribute))
- .zip(newChild.output))
- val newCond = resolvedHavingCond.transform {
- case a: AttributeReference => attrMap.getOrElse(a, a)
- }
-
- if (extraAggExprs.isEmpty) {
- Filter(newCond, newChild)
+ // `cond` might contain unresolved aggregate functions so defer its resolution to
+ // `ResolveAggregateFunctions` rule if needed.
+ if (!cond.resolved) {
Review Comment:
nit: to avoid the big diff caused by indentation, how about `if (!cond.resolved) return colResolved`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on PR #51820:
URL: https://github.com/apache/spark/pull/51820#issuecomment-3156245725
Bacport PRs: https://github.com/apache/spark/pull/51854 and https://github.com/apache/spark/pull/51855.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53094][SQL] Fix CUBE with aggregate containing HAVING clauses [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #51820:
URL: https://github.com/apache/spark/pull/51820#discussion_r2254552990
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -792,27 +792,33 @@ class Analyzer(override val catalogManager: CatalogManager) extends RuleExecutor
} else {
colResolved.havingCondition
}
- // Try resolving the condition of the filter as though it is in the aggregate clause
- val (extraAggExprs, Seq(resolvedHavingCond)) =
- ResolveAggregateFunctions.resolveExprsWithAggregate(Seq(cond), aggForResolving)
-
- // Push the aggregate expressions into the aggregate (if any).
- val newChild = constructAggregate(h, selectedGroupByExprs, groupByExprs,
- aggregate.aggregateExpressions ++ extraAggExprs, aggregate.child)
-
- // Since the output exprId will be changed in the constructed aggregate, here we build an
- // attrMap to resolve the condition again.
- val attrMap = AttributeMap((aggForResolving.output ++ extraAggExprs.map(_.toAttribute))
- .zip(newChild.output))
- val newCond = resolvedHavingCond.transform {
- case a: AttributeReference => attrMap.getOrElse(a, a)
- }
-
- if (extraAggExprs.isEmpty) {
- Filter(newCond, newChild)
+ // `cond` might contain unresolved aggregate functions so defer its resolution to
+ // `ResolveAggregateFunctions` rule if needed.
+ if (!cond.resolved) {
Review Comment:
Allright, fixed in https://github.com/apache/spark/pull/51820/commits/54ccd0feb50cb2cd3a778d07c963ef72dd192656.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org