You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "uros-b (via GitHub)" <gi...@apache.org> on 2026/07/13 13:18:57 UTC
[PR] [SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
uros-b opened a new pull request, #57222:
URL: https://github.com/apache/spark/pull/57222
### What changes were proposed in this pull request?
Note: this is a 4.1 backport of:
- https://github.com/apache/spark/pull/56891 (master/4.x)
- https://github.com/apache/spark/pull/57134 (4.2)
Lower the single-empty-grouping-set case to a global `Aggregate` (no grouping expressions, no `Expand`) in `GroupingAnalyticsTransformer`, so it returns one (grand total) row over empty input, matching the `GROUP BY`-less form and the SQL standard. `grouping_id()` folds to the constant `0`, and `grouping()`/`grouping_id()` in `HAVING`/`ORDER BY` resolve against that constant. The fix lands in both the legacy fixed-point analyzer and the single-pass resolver, which share `GroupingAnalyticsTransformer`.
### Why are the changes needed?
`GROUP BY GROUPING SETS (())` is a grand total, semantically identical to an aggregation with no `GROUP BY` clause. It was lowered to a grouped `Aggregate` over an `Expand` (grouping by `spark_grouping_id`), so over empty input it returned zero rows instead of one. The same defect affected the equivalent empty `GROUP BY CUBE()` and `GROUP BY ROLLUP()`, which also lower to a single empty grouping set.
### Does this PR introduce _any_ user-facing change?
Yes, the new (correct) behavior is gated by an internal SQL config,
`spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled` (default true). When set to false, lowering reverts to the legacy `Expand`-based form (zero rows over empty input). The flag gates all three decision points (the transformer lowering and the `grouping_id()` resolution in each analyzer) so the off state reproduces pre-fix behavior identically in both analyzers.
### How was this patch tested?
Tested via golden cases in `grouping_set.sql` (empty-input grand total, `grouping_id()` in SELECT/HAVING/ORDER BY, `grouping()` rejection, non-empty input), flag-off coverage in `grouping_set_grand_total_disabled.sql`, the regenerated `group-analytics` golden file, and `ResolveGroupingAnalyticsSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57222:
URL: https://github.com/apache/spark/pull/57222#issuecomment-4970443558
This is a long standing bug so I'm fine to fix it in 4.2+ only, and skip other backports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57222:
URL: https://github.com/apache/spark/pull/57222#issuecomment-4978947658
Thank you @dongjoon-hyun and @cloud-fan, closing this backport PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #57222:
URL: https://github.com/apache/spark/pull/57222#issuecomment-4959495335
May I ask if you already deliver this bug fix to your company product in this way, @uros-b ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57222:
URL: https://github.com/apache/spark/pull/57222#issuecomment-4963770237
> can we simply deliver this feature with `spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled=false`
We usually enable bug fixes by default, and use the SQL config for explicit opt-out (revert to incorrect behaviour). However, I don't have a strong opinion here and would be fine with disabling by default in older versions as @dongjoon-hyun suggested. @cloud-fan What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #57222: [SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input
URL: https://github.com/apache/spark/pull/57222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org