You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ulysses-you (via GitHub)" <gi...@apache.org> on 2026/07/20 04:48:13 UTC
[PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
ulysses-you opened a new pull request, #57363:
URL: https://github.com/apache/spark/pull/57363
### What changes were proposed in this pull request?
This PR enables two physical aggregate rules by default and decouples their configurations:
- `spark.sql.execution.replaceHashWithSortAgg` now defaults to `true`. It replaces a hash-based aggregate with a sort aggregate when the aggregate's child already satisfies the grouping-key sort order.
- `spark.sql.execution.combineAdjacentAggregation` now defaults to `true` and is no longer a `fallbackConf` of `replaceHashWithSortAgg`; it is an independent config. It merges an adjacent partial/final aggregate pair (with no shuffle between them) into a single complete-mode aggregate.
Affected tests and golden files are updated, and a migration-guide entry is added under "Upgrading from Spark SQL 4.2 to 4.3".
### Why are the changes needed?
Both rules improve aggregate execution and have been available but off by default. `CombineAdjacentAggregation` was introduced (SPARK-43317) as a `fallbackConf` of `replaceHashWithSortAgg`, so enabling one enabled both. They are logically independent (one reuses existing ordering, the other merges an adjacent pair), so this PR turns each on by default and gives each its own config, letting users enable/disable them separately.
### Does this PR introduce _any_ user-facing change?
Yes. On default config, Spark may now plan a sort aggregate instead of a hash aggregate (when the child is already sorted on the grouping keys) and may merge an adjacent partial/final aggregate pair into a single complete-mode aggregate. The physical plan of some queries changes as a result. To restore the previous behavior, set `spark.sql.execution.replaceHashWithSortAgg` and/or `spark.sql.execution.combineAdjacentAggregation` to `false`.
### How was this patch tested?
Updated existing tests:
- `ReplaceHashWithSortAggSuite` / `CombineAdjacentAggregationSuite`: the `checkAggs` helper now toggles both configs together; the former "falls back to replaceHashWithSortAgg" test is replaced by an independence test.
- `SQLMetricsSuite` (SPARK-25497): pins `combineAdjacentAggregation=false` so the single-partition query keeps its partial/final structure for the limit/codegen metric assertions.
- `PlannerSuite` (SPARK-40086): accepts a sort aggregate for the single-partition sorted-input queries while still asserting no extra shuffle.
- `HiveUDAFSuite` (SPARK-24935): pins `combineAdjacentAggregation=false` because the two-buffer UDAF is designed around the partial/final mode split.
Regenerated golden files via `SPARK_GENERATE_GOLDEN_FILES=1`:
- `sql-tests/results/explain-cbo.sql.out`
- `tpcds-plan-stability` approved plans for the affected TPC-DS queries.
Closes #40990
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #57363:
URL: https://github.com/apache/spark/pull/57363#issuecomment-5086425095
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/ef9cc16ed52d141dca580c0a9d8ed14e35e95ead
- merged into branch-4.x https://github.com/apache/spark/commit/00f3be78f2e452f732c7232e78a7f8d00acd998d
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you closed pull request #57363: [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default
URL: https://github.com/apache/spark/pull/57363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #57363:
URL: https://github.com/apache/spark/pull/57363#issuecomment-5057755817
Good call @wangyum, agreed. I've split the standalone merge-expression correctness fix out into #57460 (SPARK-58291), which is independent of the default flip and backportable on its own:
- `PearsonCorrelation` / `Covariance` / `CentralMomentAgg` empty-buffer merge overflow guard
- the two `DataFrameAggregateSuite` regression tests
I'll rebase this PR to drop those files once #57460 is reviewed, leaving only the default flip + config decoupling here (plus the `hiveUDFs` Complete-mode fix, which is only reachable *because* this PR enables `combineAdjacentAggregation`, so it can't be backported independently and stays here).
One ordering note: #57460 should merge first. If this PR drops the merge fix before #57460 lands, its own `DataFrameAggregateSuite` cases would reintroduce the NaN divergence they check for. So the clean sequence is merge #57460, then rebase this PR on top.
cc @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3618935409
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala:
##########
@@ -102,28 +102,36 @@ class HiveUDAFSuite extends QueryTest
test("SPARK-24935: customized Hive UDAF with two aggregation buffers") {
withTempView("v") {
spark.range(100).createTempView("v")
- val df = sql("SELECT id % 2, mock2(id) FROM v GROUP BY id % 2")
-
- val aggs = collect(df.queryExecution.executedPlan) {
- case agg: ObjectHashAggregateExec => agg
- }
-
- // There should be two aggregate operators, one for partial aggregation, and the other for
- // global aggregation.
- assert(aggs.length == 2)
-
- withSQLConf(SQLConf.OBJECT_AGG_SORT_BASED_FALLBACK_THRESHOLD.key -> "1") {
- checkAnswer(df, Seq(
- Row(0, Row(50, 0)),
- Row(1, Row(50, 0))
- ))
- }
-
- withSQLConf(SQLConf.OBJECT_AGG_SORT_BASED_FALLBACK_THRESHOLD.key -> "100") {
- checkAnswer(df, Seq(
- Row(0, Row(50, 0)),
- Row(1, Row(50, 0))
- ))
+ // `range(100)` is a single partition, so the partial and final aggregate would otherwise be
+ // adjacent and merged by `CombineAdjacentAggregation`. `MockUDAF2` deliberately uses distinct
+ // aggregation-buffer classes per aggregation mode (one for consuming original input, another
+ // for merging partial buffers), so it does not support a single `Complete`-mode aggregate.
+ // Keep the partial/final pair uncombined so the sort-based fallback path below is exercised
+ // in the mode this UDAF is designed for.
+ withSQLConf(SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> "false") {
Review Comment:
Thanks for the detailed repro. Fixed in the latest push by making the Hive wrapper Complete-safe rather than excluding UDAFs or pinning the test.
`HiveUDAFFunction` keeps two evaluators: `partial1HiveEvaluator` (PARTIAL1, buffer created in `update`) and `finalHiveEvaluator` (FINAL, buffer created in `merge`). `eval` always terminated via the FINAL evaluator, so a buffer that only went through `update` (the Complete-mode path) was a PARTIAL1-mode buffer handed to `terminate`, which mode-aware UDAFs reject.
`merge` already converted a PARTIAL1 buffer to a FINAL buffer on demand. I extracted that into `toFinalBuffer` and call it from `eval` as well, so Complete-mode (update without merge) terminates against a FINAL buffer. SPARK-24935 now runs on the default (combined) path and asserts the single Complete-mode `ObjectHashAggregateExec`, with both sort-fallback thresholds still checked. I verified a mode-aware UDAF variant no longer throws `ClassCastException`.
The change is in `sql/hive/.../hiveUDFs.scala`; `HiveUDAFSuite`, `ObjectHashAggregateSuite`, and `UDAQuerySuite` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3635086961
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -589,6 +589,29 @@ class DataFrameAggregateSuite extends SharedSparkSession
Row(null, null, null, null, null))
}
+ test("SPARK-58210: empty-buffer merge must not overflow to NaN for statistical aggregates") {
+ // A single-partition group with two equal, very large finite values has zero variance, so
+ // var_pop / covar_pop / regr_sxy must be 0.0. Previously, when adjacent Partial/Final
+ // aggregates were NOT combined (the old default), the Final merge of the non-empty Partial
+ // buffer into the empty Final buffer computed `delta * deltaN * n1 * n2` where `n1 == 0`;
+ // `delta * deltaN` overflowed to Infinity and `Infinity * 0 = NaN`, corrupting the moments.
+ // CombineAdjacentAggregation (Complete mode) sidesteps the merge and returned 0.0, so the two
+ // configurations disagreed. The merge fix makes both paths return 0.0.
+ // This must hold with and without AQE, and with combining on and off.
+ Seq(true, false).foreach { aqe =>
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString) {
+ val df = Seq(1e155, 1e155).toDF("a").repartition(1)
+ Seq(true, false).foreach { combine =>
+ withSQLConf(SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> combine.toString) {
+ checkAnswer(
+ df.selectExpr("var_pop(a)", "covar_pop(a, a)", "regr_sxy(a, a)"),
Review Comment:
Good catch -- fixed by guarding the Pearson merge the same way `Covariance`/`CentralMomentAgg` already are, rather than excluding the expressions.
`PearsonCorrelation.merge` computed `dx * dxN * n1 * n2` (and the `dy` variant) for `xMk`/`yMk` before the zero-count multiply, so when one side is an empty buffer and the other has a large average, `dx * dxN` overflowed to `Infinity` and `Infinity * 0 = NaN` corrupted the moments. I now force `dx`/`dy` to `0.0` when either side is empty and set `newXAvg`/`newYAvg` directly to the non-empty side's average, mirroring the existing fix (with a back-reference comment to `CentralMomentAgg`).
Added `SPARK-58210: empty-buffer merge must not overflow to NaN for Pearson correlation` in `DataFrameAggregateSuite`, using your exact repro (`(1.0e155, 1.0e-150), (1.000000000000001e155, 2.0e-150)`, repartitioned to 1) and asserting `corr(x, y)` and `regr_r2(y, x)` are finite and config-independent across AQE on/off x combine on/off.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala:
##########
@@ -102,15 +103,20 @@ class HiveUDAFSuite extends QueryTest
test("SPARK-24935: customized Hive UDAF with two aggregation buffers") {
withTempView("v") {
spark.range(100).createTempView("v")
+ // `range(100)` is a single partition, so `CombineAdjacentAggregation` merges the adjacent
+ // partial/final pair into a single `Complete`-mode `ObjectHashAggregateExec` (the default).
+ // `MockUDAF2` deliberately uses distinct aggregation-buffer classes per mode (one for
+ // consuming original input via PARTIAL1, another for merging partial buffers via FINAL).
+ // The Hive UDAF wrapper must convert the PARTIAL1 buffer to a FINAL buffer on `eval` so the
+ // Complete-mode path (which calls `update` but not `merge`) terminates correctly.
val df = sql("SELECT id % 2, mock2(id) FROM v GROUP BY id % 2")
Review Comment:
You're right -- the single reused DataFrame meant `executeRDD` replayed the threshold-1 execution under threshold 100, so only one path ran. Fixed: I rebuild `sql(query)` inside each `withSQLConf` block, and now assert the `numTasksFallBacked` metric -- `> 0` at threshold 1 (fallback) and `== 0` at threshold 100 (non-fallback) -- so both paths are actually exercised. Added a `numFallbackTasks` helper summing the metric across the `ObjectHashAggregateExec` nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3635087503
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala:
##########
@@ -41,11 +41,18 @@ abstract class ReplaceHashWithSortAggSuiteBase
enabledSortAggCount: Int,
disabledHashAggCount: Int,
disabledSortAggCount: Int): Unit = {
- withSQLConf(SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "true") {
+ // Toggle both rules together so the "disabled" case keeps the raw partial/final aggregate
+ // structure (the queries here are designed around `ReplaceHashWithSortAgg`, and an enabled
+ // `CombineAdjacentAggregation` would merge adjacent partial/final pairs and change the counts).
+ withSQLConf(
+ SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "true",
+ SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> "true") {
Review Comment:
Agreed, the together case enabled combining and collapsed the pair before traversal, so it couldn't detect reverting `transformUp`. Added `SPARK-58210: bottom-up traversal replaces both partial and final hash aggregate` with `replaceHashWithSortAgg=true` and `combineAdjacentAggregation=false`, over the SMJ -> partial -> final shape, asserting 0 hash / 2 sort.
I ran the same mutation you did: reverting `transformUp` -> `transformDown` now fails this test in both AQE modes (1 hash agg remains instead of 0), and it passes once reverted back. So the traversal change is now genuinely covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #57363:
URL: https://github.com/apache/spark/pull/57363#issuecomment-5029517899
Thanks @viirya for the thorough trace — agreed on both fixes. Re: `transformDown` -> `transformUp`: good call, I've added a bullet to the PR description explaining it. The reasoning matches what you laid out: `HashAggregateExec` does not expose a grouping-key `outputOrdering` (`Nil`), while `SortAggregateExec` is order-preserving on the grouping keys. With `transformDown` an outer aggregate inspected before its inner partial aggregate sees the partial's `Nil` ordering and is not replaced; with `transformUp` the partial is replaced first, so its sort-aggregate output ordering satisfies the outer aggregate, which is then also replaced. Each replacement is still gated by the `orderingSatisfies` check, so bottom-up can only enable more valid replacements, never wrong ones. The "replace partial and final hash aggregate together with sort aggregate" case in `ReplaceHashWithSortAggSuite` (SMJ -> partial -> final, expecting both replaced) covers it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3634650842
##########
sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala:
##########
@@ -589,6 +589,29 @@ class DataFrameAggregateSuite extends SharedSparkSession
Row(null, null, null, null, null))
}
+ test("SPARK-58210: empty-buffer merge must not overflow to NaN for statistical aggregates") {
+ // A single-partition group with two equal, very large finite values has zero variance, so
+ // var_pop / covar_pop / regr_sxy must be 0.0. Previously, when adjacent Partial/Final
+ // aggregates were NOT combined (the old default), the Final merge of the non-empty Partial
+ // buffer into the empty Final buffer computed `delta * deltaN * n1 * n2` where `n1 == 0`;
+ // `delta * deltaN` overflowed to Infinity and `Infinity * 0 = NaN`, corrupting the moments.
+ // CombineAdjacentAggregation (Complete mode) sidesteps the merge and returned 0.0, so the two
+ // configurations disagreed. The merge fix makes both paths return 0.0.
+ // This must hold with and without AQE, and with combining on and off.
+ Seq(true, false).foreach { aqe =>
+ withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString) {
+ val df = Seq(1e155, 1e155).toDF("a").repartition(1)
+ Seq(true, false).foreach { combine =>
+ withSQLConf(SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> combine.toString) {
+ checkAnswer(
+ df.selectExpr("var_pop(a)", "covar_pop(a, a)", "regr_sxy(a, a)"),
Review Comment:
[P2] Please extend this empty-buffer fix and AQE-by-combining regression to corr and regr_r2 as well. PearsonCorrelation.merge still evaluates dx * dxN * n1 * n2 and dy * dyN * n1 * n2 before the zero-count multiplication, so the same Infinity * 0 corruption survives this change. I reproduced this at this exact head with AQE disabled and Seq((1.0e155, 1.0e-150), (1.000000000000001e155, 2.0e-150)).toDF("x", "y").repartition(1).selectExpr("corr(x, y)", "regr_r2(y, x)"): with combineAdjacentAggregation=true, the result is (1.0, 1.0000000000000004); with it disabled, both results are NaN. Since this PR enables combining by default, a physical-optimization toggle still changes the observable answers. Please guard the Pearson empty-buffer merge as well and cover both expressions across AQE on/off and combining on/off.
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala:
##########
@@ -41,11 +41,18 @@ abstract class ReplaceHashWithSortAggSuiteBase
enabledSortAggCount: Int,
disabledHashAggCount: Int,
disabledSortAggCount: Int): Unit = {
- withSQLConf(SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "true") {
+ // Toggle both rules together so the "disabled" case keeps the raw partial/final aggregate
+ // structure (the queries here are designed around `ReplaceHashWithSortAgg`, and an enabled
+ // `CombineAdjacentAggregation` would merge adjacent partial/final pairs and change the counts).
+ withSQLConf(
+ SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "true",
+ SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> "true") {
Review Comment:
[P3] Please add a traversal regression with replaceHashWithSortAgg=true and combineAdjacentAggregation=false, and assert that both the partial and final nodes become SortAggregateExec. This helper currently enables combining together with replacement, so the partial/final pair is collapsed before the test can establish that bottom-up traversal replaces both nodes; the together case therefore observes only one sort aggregate. I mutation-tested the exact head by reverting transformUp to transformDown: both the normal and AQE together tests still passed. Consequently the test cited as coverage for the traversal change cannot detect reverting that change. A sorted-input two-node case with combining disabled, run with AQE both on and off, would fail under transformDown and cover the behavior.
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala:
##########
@@ -102,15 +103,20 @@ class HiveUDAFSuite extends QueryTest
test("SPARK-24935: customized Hive UDAF with two aggregation buffers") {
withTempView("v") {
spark.range(100).createTempView("v")
+ // `range(100)` is a single partition, so `CombineAdjacentAggregation` merges the adjacent
+ // partial/final pair into a single `Complete`-mode `ObjectHashAggregateExec` (the default).
+ // `MockUDAF2` deliberately uses distinct aggregation-buffer classes per mode (one for
+ // consuming original input via PARTIAL1, another for merging partial buffers via FINAL).
+ // The Hive UDAF wrapper must convert the PARTIAL1 buffer to a FINAL buffer on `eval` so the
+ // Complete-mode path (which calls `update` but not `merge`) terminates correctly.
val df = sql("SELECT id % 2, mock2(id) FROM v GROUP BY id % 2")
Review Comment:
[P3] Please construct a fresh query inside each fallback-threshold withSQLConf block. This one DataFrame is executed first with threshold 1, and SparkPlan.executeRDD memoizes the resulting RDD; the subsequent checkAnswer(df, ...) under threshold 100 reuses that already-created execution instead of constructing an ObjectHashAggregateExec execution under the new threshold. I verified that both checks use the same execution-RDD ID, so this does not exercise both fallback and non-fallback despite the test's claim. ObjectHashAggregateSuite also explicitly cautions against reusing a DataFrame while changing this threshold. Rebuilding sql(...) inside each configuration, and ideally asserting the fallback metric, would make the lifecycle fix actually cover both paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3616717627
##########
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveUDAFSuite.scala:
##########
@@ -102,28 +102,36 @@ class HiveUDAFSuite extends QueryTest
test("SPARK-24935: customized Hive UDAF with two aggregation buffers") {
withTempView("v") {
spark.range(100).createTempView("v")
- val df = sql("SELECT id % 2, mock2(id) FROM v GROUP BY id % 2")
-
- val aggs = collect(df.queryExecution.executedPlan) {
- case agg: ObjectHashAggregateExec => agg
- }
-
- // There should be two aggregate operators, one for partial aggregation, and the other for
- // global aggregation.
- assert(aggs.length == 2)
-
- withSQLConf(SQLConf.OBJECT_AGG_SORT_BASED_FALLBACK_THRESHOLD.key -> "1") {
- checkAnswer(df, Seq(
- Row(0, Row(50, 0)),
- Row(1, Row(50, 0))
- ))
- }
-
- withSQLConf(SQLConf.OBJECT_AGG_SORT_BASED_FALLBACK_THRESHOLD.key -> "100") {
- checkAnswer(df, Seq(
- Row(0, Row(50, 0)),
- Row(1, Row(50, 0))
- ))
+ // `range(100)` is a single partition, so the partial and final aggregate would otherwise be
+ // adjacent and merged by `CombineAdjacentAggregation`. `MockUDAF2` deliberately uses distinct
+ // aggregation-buffer classes per aggregation mode (one for consuming original input, another
+ // for merging partial buffers), so it does not support a single `Complete`-mode aggregate.
+ // Keep the partial/final pair uncombined so the sort-based fallback path below is exercised
+ // in the mode this UDAF is designed for.
+ withSQLConf(SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> "false") {
Review Comment:
[P1] Please do not disable this regression test to accommodate the new default. A Complete object aggregate calls `HiveUDAFFunction.update`, which creates a PARTIAL1-mode buffer, and then `eval`, which passes that buffer to the separately initialized FINAL evaluator. Mode-aware Hive UDAFs may legally use different buffer classes in those modes, so the default rewrite can turn a previously supported query into a `ClassCastException`. I reproduced this and also verified that a variant which correctly handles Hive's native COMPLETE lifecycle still fails through Spark's mixed-evaluator path. Please make the Hive wrapper Complete-safe or exclude these UDAFs before enabling the rule by default, and keep this test exercising the default path.
##########
docs/sql-migration-guide.md:
##########
@@ -28,6 +28,7 @@ license: |
- Since Spark 4.3, the configuration key `spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled` has been renamed to `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` to reflect that it now applies to storage-partitioned joins, aggregates, and windows. The old key continues to work as an alias.
- Since Spark 4.3, the Spark Thrift Server rejects setting JVM system properties through the `set:system:` session configuration overlay (for example, in a JDBC connection string). To restore the previous behavior, set `spark.sql.legacy.hive.thriftServer.allowSettingSystemProperties` to `true`.
- Since Spark 4.3, the adaptive execution rule `org.apache.spark.sql.execution.adaptive.DynamicJoinSelection` has been renamed to `DemoteBroadcastHashJoin`, which now only demotes broadcast hash joins (emitting `NO_BROADCAST_HASH`). Its selection of shuffled hash join over sort merge join has moved to a new physical rule gated by `spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled` (default `true`). If you previously disabled the shuffled-hash-join preference by listing `org.apache.spark.sql.execution.adaptive.DynamicJoinSelection` in `spark.sql.adaptive.optimizer.excludedRules`, that name no longer matches any rule (unknown names are silently ignored); set `spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled` to `false` instead.
+- Since Spark 4.3, `spark.sql.execution.replaceHashWithSortAgg` defaults to `true`. Spark now replaces a hash-based aggregate with a sort aggregate when the aggregate's child is already sorted on the grouping keys. To restore the previous behavior, set `spark.sql.execution.replaceHashWithSortAgg` to `false`.
Review Comment:
[P2] Please document `spark.sql.execution.combineAdjacentAggregation` here as well. Although the config itself is introduced during 4.3 development, its default-on execution behavior is new for users upgrading from 4.2. The two settings are now independent, so following this rollback instruction and disabling only `replaceHashWithSortAgg` still leaves adjacent Partial/Final aggregation combined. Please restore the removed entry and state that both settings may need to be false to recover the previous staging.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3006,18 +3006,17 @@ object SQLConf {
.doc("Whether to replace hash aggregate node with sort aggregate based on children's ordering")
.version("3.3.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
val COMBINE_ADJACENT_AGGREGATION_ENABLED =
buildConf("spark.sql.execution.combineAdjacentAggregation")
.internal()
.doc("When true, combine adjacent aggregation with `Partial` and `Final` to `Complete` " +
- "mode. This defaults to the value of `spark.sql.execution.replaceHashWithSortAgg` since " +
- "combining adjacent aggregation subsumes the partial-and-final merge that " +
- "`replaceHashWithSortAgg` used to perform on its own.")
+ "mode.")
.version("4.3.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
- .fallbackConf(REPLACE_HASH_WITH_SORT_AGG_ENABLED)
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
[P2] Enabling this rewrite changes built-in aggregate results, not only the physical plan. With AQE off and a single-partition group containing two finite `1e155` values, disabling combining returns `NaN` for `var_pop`, `covar_pop`, and `regr_sxy`, while Complete returns `0.0`. The old NaN comes from overflowing the first merge into the empty Final buffer, so 0.0 is mathematically preferable, but a default physical optimization should not silently introduce config-dependent results. Please fix and test the empty-buffer merge as a deliberate correctness change, or keep these aggregates out of combining until the modes are equivalent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #57363:
URL: https://github.com/apache/spark/pull/57363#issuecomment-5020356784
cc @viirya @cloud-fan @sunchao @wangyum thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3618936658
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -3006,18 +3006,17 @@ object SQLConf {
.doc("Whether to replace hash aggregate node with sort aggregate based on children's ordering")
.version("3.3.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
val COMBINE_ADJACENT_AGGREGATION_ENABLED =
buildConf("spark.sql.execution.combineAdjacentAggregation")
.internal()
.doc("When true, combine adjacent aggregation with `Partial` and `Final` to `Complete` " +
- "mode. This defaults to the value of `spark.sql.execution.replaceHashWithSortAgg` since " +
- "combining adjacent aggregation subsumes the partial-and-final merge that " +
- "`replaceHashWithSortAgg` used to perform on its own.")
+ "mode.")
.version("4.3.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
- .fallbackConf(REPLACE_HASH_WITH_SORT_AGG_ENABLED)
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
Great catch, and thanks for pinpointing the empty-buffer merge. Fixed as a deliberate correctness change in the merge expressions rather than excluding the aggregates.
Root cause: `CentralMomentAgg.merge` computes `delta * deltaN * n1 * n2` and `Covariance.merge` computes `dx * dyN * n1 * n2`. When one side is an empty buffer (`n1 == 0` or `n2 == 0`) and the other side has a large average, `delta * deltaN` (or `dx * dyN`) overflows to `Infinity` before being multiplied by the zero count, and `Infinity * 0 = NaN` corrupts the merged moments. The existing `newN === 0` guard only covers both-empty. The old default (no combining) hit this on a single-partition group of two large equal values, returning `NaN`; the Complete-mode default sidestepped the merge and returned the correct `0.0`, so the two configs disagreed.
Fix: force `delta`/`dx`/`dy` to `0.0` when either side is empty, and set `newAvg`/`newXAvg`/`newYAvg` directly to the non-empty side's average. The delta terms then vanish cleanly and the merged buffer equals the non-empty side. Both the combined and the separate paths now return `0.0` for `var_pop`/`covar_pop`/`regr_sxy`.
Added `SPARK-58210: empty-buffer merge must not overflow to NaN for statistical aggregates` in `DataFrameAggregateSuite`, covering `var_pop`, `covar_pop`, `regr_sxy` across AQE on/off x combine on/off. The `group-by`, `linear-regression`, and `aggregates_part1` SQL golden files are unchanged (regenerated to confirm).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #57363:
URL: https://github.com/apache/spark/pull/57363#issuecomment-5057097838
Is it possible splitting this PR into two before merging, because it currently bundles two logically independent changes:
1. The merge-expression correctness fix (standalone bug).
2. The default flip + config decoupling
Bundling them makes the merge fix impossible to backport to 4.2.x / 3.5.x. If we later find a regression from the default flip and want to revert just that, the correctness fix would go with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3618938565
##########
docs/sql-migration-guide.md:
##########
@@ -28,6 +28,7 @@ license: |
- Since Spark 4.3, the configuration key `spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled` has been renamed to `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` to reflect that it now applies to storage-partitioned joins, aggregates, and windows. The old key continues to work as an alias.
- Since Spark 4.3, the Spark Thrift Server rejects setting JVM system properties through the `set:system:` session configuration overlay (for example, in a JDBC connection string). To restore the previous behavior, set `spark.sql.legacy.hive.thriftServer.allowSettingSystemProperties` to `true`.
- Since Spark 4.3, the adaptive execution rule `org.apache.spark.sql.execution.adaptive.DynamicJoinSelection` has been renamed to `DemoteBroadcastHashJoin`, which now only demotes broadcast hash joins (emitting `NO_BROADCAST_HASH`). Its selection of shuffled hash join over sort merge join has moved to a new physical rule gated by `spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled` (default `true`). If you previously disabled the shuffled-hash-join preference by listing `org.apache.spark.sql.execution.adaptive.DynamicJoinSelection` in `spark.sql.adaptive.optimizer.excludedRules`, that name no longer matches any rule (unknown names are silently ignored); set `spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled` to `false` instead.
+- Since Spark 4.3, `spark.sql.execution.replaceHashWithSortAgg` defaults to `true`. Spark now replaces a hash-based aggregate with a sort aggregate when the aggregate's child is already sorted on the grouping keys. To restore the previous behavior, set `spark.sql.execution.replaceHashWithSortAgg` to `false`.
Review Comment:
Good point — restored. The migration guide now has a separate entry for `spark.sql.execution.combineAdjacentAggregation` under "Upgrading from Spark SQL 4.2 to 4.3", and it states explicitly that the setting is independent of `replaceHashWithSortAgg`, so disabling only `replaceHashWithSortAgg` still leaves adjacent aggregation combined; both must be set to `false` to fully restore the previous partial/final staging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57363:
URL: https://github.com/apache/spark/pull/57363#discussion_r3646716049
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/CombineAdjacentAggregationSuite.scala:
##########
@@ -282,27 +282,35 @@ class CombineAdjacentAggregationSuite extends QueryTest
}
}
- test("Default value falls back to spark.sql.execution.replaceHashWithSortAgg") {
+ test("combineAdjacentAggregation is independent of replaceHashWithSortAgg") {
withTempView("t") {
spark.range(20).selectExpr("id % 3 as k", "id % 7 as v")
.createOrReplaceTempView("t")
val query = "SELECT k, count(*) FROM (SELECT /*+ repartition(k) */ * FROM t) GROUP BY k"
- // `spark.sql.execution.combineAdjacentAggregation` is unset here, so it falls back to
- // `spark.sql.execution.replaceHashWithSortAgg`.
- withSQLConf(SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "false") {
- assert(numAggregates(query) == 2)
- }
- withSQLConf(SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "true") {
+ // `spark.sql.execution.combineAdjacentAggregation` has its own default and does not fall
+ // back to `spark.sql.execution.replaceHashWithSortAgg`, so each can be toggled on its own.
+ // Combining is driven solely by `combineAdjacentAggregation` here: the partial and final
+ // aggregate are adjacent (the input is already partitioned by `k`), so enabling it merges
Review Comment:
The subject names two stages, so this should be plural.
```suggestion
// aggregates are adjacent (the input is already partitioned by `k`), so enabling it merges
```
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/ReplaceHashWithSortAggSuite.scala:
##########
@@ -96,6 +103,51 @@ abstract class ReplaceHashWithSortAggSuiteBase
}
}
+ test("SPARK-58210: bottom-up traversal replaces both partial and final hash aggregate") {
+ // Regression test for the `transformDown` -> `transformUp` change. With combining disabled the
+ // partial/final pair is NOT collapsed, so both hash aggregates remain over a sort merge join
+ // whose output is already ordered on the grouping key. `HashAggregateExec` exposes no
+ // grouping-key `outputOrdering` (`Nil`) while `SortAggregateExec` is order-preserving, so the
+ // final aggregate's replacement depends on its child (the partial aggregate) having been
+ // replaced first. Bottom-up traversal replaces the partial aggregate, whose sort-aggregate
+ // output ordering then satisfies the final aggregate, which is replaced in turn: 0 hash, 2
+ // sort. Top-down traversal would inspect the final aggregate before the partial is replaced,
+ // see the partial's `Nil` ordering, and leave the final as a hash aggregate: 1 hash, 1 sort.
+ // The "together" test above cannot catch this because it enables combining, which collapses
+ // the pair into a single node before traversal.
+ withTempView("t1", "t2") {
+ spark.range(100).selectExpr("id as key").createOrReplaceTempView("t1")
+ spark.range(50).selectExpr("id as key").createOrReplaceTempView("t2")
+ Seq("COUNT", "COLLECT_LIST").foreach { aggExpr =>
+ val query =
+ s"""
+ |SELECT key, $aggExpr(key)
+ |FROM
+ |(
+ | SELECT /*+ SHUFFLE_MERGE(t1) */ t1.key AS key
+ | FROM t1
+ | JOIN t2
+ | ON t1.key = t2.key
+ |)
+ |GROUP BY key
+ """.stripMargin
+ val expected = withSQLConf(
+ SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "false",
+ SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> "false") {
+ sql(query).collect()
+ }
+ withSQLConf(
+ SQLConf.REPLACE_HASH_WITH_SORT_AGG_ENABLED.key -> "true",
+ SQLConf.COMBINE_ADJACENT_AGGREGATION_ENABLED.key -> "false") {
+ val df = sql(query)
+ // Both the partial and final hash aggregate are replaced with sort aggregates.
Review Comment:
The subject names two stages, so this should be plural.
```suggestion
// Both the partial and final hash aggregates are replaced with sort aggregates.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58210][SQL] Enable ReplaceHashWithSortAgg and CombineAdjacentAggregation by default [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on PR #57363:
URL: https://github.com/apache/spark/pull/57363#issuecomment-5086427370
Thank you all for review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org