You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "uros-db (via GitHub)" <gi...@apache.org> on 2026/06/30 06:41:53 UTC
[PR] [WIP][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
uros-db opened a new pull request, #56891:
URL: https://github.com/apache/spark/pull/56891
### What changes were proposed in this pull request?
Lower the single-empty-grouping-set case to a global `Aggregate` (no grouping expressions, no `Expand`) in `GroupingAnalyticsTransformer`, so it returns one (grand total) row over empty input, matching the `GROUP BY`-less form and the SQL standard. `grouping_id()` folds to the constant `0`, and `grouping()`/`grouping_id()` in `HAVING`/`ORDER BY` resolve against that constant. The fix lands in both the legacy fixed-point analyzer and the single-pass resolver, which share `GroupingAnalyticsTransformer`.
### Why are the changes needed?
`GROUP BY GROUPING SETS (())` is a grand total, semantically identical to an aggregation with no `GROUP BY` clause. It was lowered to a grouped `Aggregate` over an `Expand` (grouping by `spark_grouping_id`), so over empty input it returned zero rows instead of one. The same defect affected the equivalent empty `GROUP BY CUBE()` and `GROUP BY ROLLUP()`, which also lower to a single empty grouping set.
### Does this PR introduce _any_ user-facing change?
Yes, the new (correct) behavior is gated by an internal SQL config,
`spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled` (default true). When set to false, lowering reverts to the legacy `Expand`-based form (zero rows over empty input). The flag gates all three decision points (the transformer lowering and the `grouping_id()` resolution in each analyzer) so the off state reproduces pre-fix behavior identically in both analyzers.
### How was this patch tested?
Tested via golden cases in `grouping_set.sql` (empty-input grand total, `grouping_id()` in SELECT/HAVING/ORDER BY, `grouping()` rejection, non-empty input), flag-off coverage in `grouping_set_grand_total_disabled.sql`, the regenerated `group-analytics` golden file, and `ResolveGroupingAnalyticsSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #56891: [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input
URL: https://github.com/apache/spark/pull/56891
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #56891:
URL: https://github.com/apache/spark/pull/56891#issuecomment-4909464700
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/653eb498d077ac380a007ccc4b1ea97ec95d71a7
- merged into branch-4.x https://github.com/apache/spark/commit/65d9c4f7ab3c3d6c75df279772fa91ffc7c7d001
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #56891:
URL: https://github.com/apache/spark/pull/56891#discussion_r3528833549
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/GroupingAnalyticsTransformer.scala:
##########
@@ -153,19 +243,27 @@ object GroupingAnalyticsTransformer extends SQLConfHelper with AliasHelper {
/**
* Collect the last grouping expression since the provided [[Aggregate]] should have grouping id
* as the last grouping key.
+ *
+ * A grand-total `GROUP BY GROUPING SETS (())` is lowered to a global [[Aggregate]] with no
+ * grouping expressions (see [[apply]]), so there is no `spark_grouping_id` key and no grouping
+ * columns; return an empty sequence for it.
*/
def collectGroupingExpressions(aggregate: Aggregate): Seq[Expression] = {
- val gid = aggregate.groupingExpressions.last
- gid match {
- case attributeReference: AttributeReference =>
- if (attributeReference.name != VirtualColumn.groupingIdName) {
+ if (aggregate.groupingExpressions.isEmpty) {
Review Comment:
Thank you Max! Confirmed - real bug, fixed in 14544d66880. Now tag the lowered grand total (GRAND_TOTAL_AGGREGATE_TAG) and key collectGroupingExpressions on the tag, so a plain no-GROUP BY aggregate still throws UNSUPPORTED_GROUPING_EXPRESSION (parity with the SELECT-list path). Added negative golden cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #56891:
URL: https://github.com/apache/spark/pull/56891#discussion_r3528838573
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -132,6 +132,17 @@ class DataFrameAggregateSuite extends SharedSparkSession
)
}
+ test("cube()/rollup() with no grouping columns return one grand-total row over empty input") {
+ // With no grouping columns, cube()/rollup() lower to a global aggregate (the grand total),
+ // which returns one row even over empty input -- like an aggregation with no GROUP BY clause.
+ // This is the DataFrame-API surface for the empty CUBE/ROLLUP case (not expressible in SQL).
+ checkAnswer(spark.range(0).cube().count(), Row(0L))
Review Comment:
Added to ClientE2ETestSuite (14544d66880); validated by Connect CI since it's a RemoteSparkSession E2E test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on code in PR #56891:
URL: https://github.com/apache/spark/pull/56891#discussion_r3528836050
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/GroupingAnalyticsTransformer.scala:
##########
@@ -81,34 +88,117 @@ object GroupingAnalyticsTransformer extends SQLConfHelper with AliasHelper {
child: LogicalPlan,
aggregationExpressions: Seq[NamedExpression]): Aggregate = {
- val groupByAliases = constructGroupByAlias(newAlias, groupByExpressions)
+ if (shouldLowerToGrandTotalAggregate(groupByExpressions, selectedGroupByExpressions)) {
+ constructGrandTotalAggregate(newAlias, aggregationExpressions, child)
+ } else {
+ val groupByAliases = constructGroupByAlias(newAlias, groupByExpressions)
- val gid = AttributeReference(VirtualColumn.groupingIdName, GroupingID.dataType, false)()
- val expand = constructExpand(
- selectedGroupByExpressions = selectedGroupByExpressions,
- child = child,
- groupByAliases = groupByAliases,
- gid = gid,
- childOutput = childOutput
- )
- val groupingAttributes = expand.output.drop(childOutput.length)
+ val gid = AttributeReference(VirtualColumn.groupingIdName, GroupingID.dataType, false)()
+ val expand = constructExpand(
+ selectedGroupByExpressions = selectedGroupByExpressions,
+ child = child,
+ groupByAliases = groupByAliases,
+ gid = gid,
+ childOutput = childOutput
+ )
+ val groupingAttributes = expand.output.drop(childOutput.length)
- val aggregations = constructAggregateExpressions(
- newAlias = newAlias,
- groupByExpressions = groupByExpressions,
- aggregations = aggregationExpressions,
- groupByAliases = groupByAliases,
- groupingAttributes = groupingAttributes,
- gid = gid
- )
+ val aggregations = constructAggregateExpressions(
+ newAlias = newAlias,
+ groupByExpressions = groupByExpressions,
+ aggregations = aggregationExpressions,
+ groupByAliases = groupByAliases,
+ groupingAttributes = groupingAttributes,
+ gid = gid
+ )
+
+ Aggregate(
+ groupingExpressions = groupingAttributes,
+ aggregateExpressions = aggregations,
+ child = expand
+ )
+ }
+ }
- val aggregate = Aggregate(
- groupingExpressions = groupingAttributes,
+ /**
+ * Whether a grouping-set spec is the grand-total-only case `GROUP BY GROUPING SETS (())` (and
+ * the equivalent empty `CUBE()`/`ROLLUP()`) that [[apply]] lowers to a global [[Aggregate]]:
+ * no leading group-by expressions and a single empty grouping set, with
+ * [[SQLConf.LOWER_EMPTY_GROUPING_SET_TO_GLOBAL_AGGREGATE]] enabled. This is the
+ * pre-lowering decision; [[isLoweredToGrandTotalAggregate]] detects the same case post-lowering.
+ */
+ def shouldLowerToGrandTotalAggregate(
+ groupByExpressions: Seq[Expression],
+ selectedGroupByExpressions: Seq[Seq[Expression]]): Boolean = {
+ conf.getConf(SQLConf.LOWER_EMPTY_GROUPING_SET_TO_GLOBAL_AGGREGATE) &&
+ groupByExpressions.isEmpty &&
+ selectedGroupByExpressions.length == 1 &&
+ selectedGroupByExpressions.head.isEmpty
+ }
+
+ /**
+ * Whether a lowered [[Aggregate]] is the grand total produced by
+ * [[shouldLowerToGrandTotalAggregate]]: its resolved grouping expressions (without the
+ * `spark_grouping_id` key, as returned by [[collectGroupingExpressions]]) are empty, with the
+ * flag enabled. The flag is part of the check because with it off the same grand total is
+ * lowered via [[Expand]], whose `spark_grouping_id` key must still be referenced.
+ *
+ * This keys purely on the collected grouping expressions being empty, so it also matches a
+ * value-equivalent all-empty multi-set [[Expand]] aggregate (e.g. `GROUP BY GROUPING SETS ((),
Review Comment:
Added (SELECT/HAVING/ORDER BY, flag-on/off). Note: ((), ()) is a duplicated set → gets _gen_grouping_pos, stays on Expand, so it's not lowered and HAVING/ORDER BY errors already on master. Corrected the scaladoc/comments accordingly. (14544d66880)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-db (via GitHub)" <gi...@apache.org>.
uros-db commented on PR #56891:
URL: https://github.com/apache/spark/pull/56891#issuecomment-4892768104
@MaxGekk On the MISSING_AGGREGATION → MISSING_GROUP_BY change: this is intentional and emergent from the fix rather than a deliberate message choice. Once GROUP BY GROUPING SETS (()) lowers to a global Aggregate with no grouping expressions, the plan is structurally identical to a no-GROUP BY aggregation, so a non-aggregated select column takes the same MISSING_GROUP_BY path that SELECT c1 FROM t (no GROUP BY) would. I agree the "Add GROUP BY" remedy reads slightly oddly given the query does have a GROUP BY clause, but the new error is arguably more accurate about the underlying problem (the grand total groups by nothing), and it matches the no-GROUP BY form the PR is deliberately aligning with.
All other comments addressed, thank you for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56891:
URL: https://github.com/apache/spark/pull/56891#discussion_r3500534995
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -132,6 +132,17 @@ class DataFrameAggregateSuite extends SharedSparkSession
)
}
+ test("cube()/rollup() with no grouping columns return one grand-total row over empty input") {
+ // With no grouping columns, cube()/rollup() lower to a global aggregate (the grand total),
+ // which returns one row even over empty input -- like an aggregation with no GROUP BY clause.
+ // This is the DataFrame-API surface for the empty CUBE/ROLLUP case (not expressible in SQL).
+ checkAnswer(spark.range(0).cube().count(), Row(0L))
Review Comment:
Good call covering empty `cube()`/`rollup()` here — it's the only reachable surface, since the grammar requires ≥1 grouping set inside `CUBE(...)`/`ROLLUP(...)`, so empty `CUBE()`/`ROLLUP()` isn't SQL-expressible. Optional: a Spark Connect DataFrame case would additionally exercise the Connect analyzed-plan path, which differs from classic.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/GroupingAnalyticsTransformer.scala:
##########
@@ -81,34 +88,117 @@ object GroupingAnalyticsTransformer extends SQLConfHelper with AliasHelper {
child: LogicalPlan,
aggregationExpressions: Seq[NamedExpression]): Aggregate = {
- val groupByAliases = constructGroupByAlias(newAlias, groupByExpressions)
+ if (shouldLowerToGrandTotalAggregate(groupByExpressions, selectedGroupByExpressions)) {
+ constructGrandTotalAggregate(newAlias, aggregationExpressions, child)
+ } else {
+ val groupByAliases = constructGroupByAlias(newAlias, groupByExpressions)
- val gid = AttributeReference(VirtualColumn.groupingIdName, GroupingID.dataType, false)()
- val expand = constructExpand(
- selectedGroupByExpressions = selectedGroupByExpressions,
- child = child,
- groupByAliases = groupByAliases,
- gid = gid,
- childOutput = childOutput
- )
- val groupingAttributes = expand.output.drop(childOutput.length)
+ val gid = AttributeReference(VirtualColumn.groupingIdName, GroupingID.dataType, false)()
+ val expand = constructExpand(
+ selectedGroupByExpressions = selectedGroupByExpressions,
+ child = child,
+ groupByAliases = groupByAliases,
+ gid = gid,
+ childOutput = childOutput
+ )
+ val groupingAttributes = expand.output.drop(childOutput.length)
- val aggregations = constructAggregateExpressions(
- newAlias = newAlias,
- groupByExpressions = groupByExpressions,
- aggregations = aggregationExpressions,
- groupByAliases = groupByAliases,
- groupingAttributes = groupingAttributes,
- gid = gid
- )
+ val aggregations = constructAggregateExpressions(
+ newAlias = newAlias,
+ groupByExpressions = groupByExpressions,
+ aggregations = aggregationExpressions,
+ groupByAliases = groupByAliases,
+ groupingAttributes = groupingAttributes,
+ gid = gid
+ )
+
+ Aggregate(
+ groupingExpressions = groupingAttributes,
+ aggregateExpressions = aggregations,
+ child = expand
+ )
+ }
+ }
- val aggregate = Aggregate(
- groupingExpressions = groupingAttributes,
+ /**
+ * Whether a grouping-set spec is the grand-total-only case `GROUP BY GROUPING SETS (())` (and
+ * the equivalent empty `CUBE()`/`ROLLUP()`) that [[apply]] lowers to a global [[Aggregate]]:
+ * no leading group-by expressions and a single empty grouping set, with
+ * [[SQLConf.LOWER_EMPTY_GROUPING_SET_TO_GLOBAL_AGGREGATE]] enabled. This is the
+ * pre-lowering decision; [[isLoweredToGrandTotalAggregate]] detects the same case post-lowering.
+ */
+ def shouldLowerToGrandTotalAggregate(
+ groupByExpressions: Seq[Expression],
+ selectedGroupByExpressions: Seq[Seq[Expression]]): Boolean = {
+ conf.getConf(SQLConf.LOWER_EMPTY_GROUPING_SET_TO_GLOBAL_AGGREGATE) &&
+ groupByExpressions.isEmpty &&
+ selectedGroupByExpressions.length == 1 &&
+ selectedGroupByExpressions.head.isEmpty
+ }
+
+ /**
+ * Whether a lowered [[Aggregate]] is the grand total produced by
+ * [[shouldLowerToGrandTotalAggregate]]: its resolved grouping expressions (without the
+ * `spark_grouping_id` key, as returned by [[collectGroupingExpressions]]) are empty, with the
+ * flag enabled. The flag is part of the check because with it off the same grand total is
+ * lowered via [[Expand]], whose `spark_grouping_id` key must still be referenced.
+ *
+ * This keys purely on the collected grouping expressions being empty, so it also matches a
+ * value-equivalent all-empty multi-set [[Expand]] aggregate (e.g. `GROUP BY GROUPING SETS ((),
Review Comment:
This `((),())` value-equivalence is the right call (two empty sets stay on the Expand path, but every Expand row has `spark_grouping_id = 0`, so folding `grouping_id()` to `0` matches — and both analyzers fold identically via the shared helpers, so no divergence). It's just untested: worth a golden case for `GROUP BY GROUPING SETS ((), ())` with `grouping_id()` in SELECT/HAVING/ORDER BY, flag-on and flag-off, to lock the documented invariant down.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/GroupingAnalyticsTransformer.scala:
##########
@@ -153,19 +243,27 @@ object GroupingAnalyticsTransformer extends SQLConfHelper with AliasHelper {
/**
* Collect the last grouping expression since the provided [[Aggregate]] should have grouping id
* as the last grouping key.
+ *
+ * A grand-total `GROUP BY GROUPING SETS (())` is lowered to a global [[Aggregate]] with no
+ * grouping expressions (see [[apply]]), so there is no `spark_grouping_id` key and no grouping
+ * columns; return an empty sequence for it.
*/
def collectGroupingExpressions(aggregate: Aggregate): Seq[Expression] = {
- val gid = aggregate.groupingExpressions.last
- gid match {
- case attributeReference: AttributeReference =>
- if (attributeReference.name != VirtualColumn.groupingIdName) {
+ if (aggregate.groupingExpressions.isEmpty) {
Review Comment:
With this guard returning `Seq.empty`, and `isLoweredToGrandTotalAggregate` keying purely on emptiness, a lowered grand total becomes indistinguishable from a plain no-`GROUP BY` aggregate (both have empty `groupingExpressions`). The `Filter`/`Sort` grouping-function cases in `Analyzer` fire on `hasGroupingFunction(cond)` with no child-type guard, so `grouping_id()`/`grouping()` in HAVING/ORDER BY over a plain aggregate — e.g. `SELECT count(*) FROM t HAVING grouping_id() = 0` — appears to fold to `Literal(0)` and succeed here, whereas the SELECT-list form errors `UNSUPPORTED_GROUPING_EXPRESSION` via `CheckAnalysis` (and pre-PR this path hit `Seq.last` on empty). Could you confirm the behavior? A negative golden case (a grouping function in HAVING and in ORDER BY with no grouping set) would pin it down; if it does fold, a child-type guard so a plain aggregate still rejects it would keep parity with the SELECT-list path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org