You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "stojkomilos (via GitHub)" <gi...@apache.org> on 2026/10/15 12:19:22 UTC
[PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
stojkomilos opened a new pull request, #59507:
URL: https://github.com/apache/spark/pull/59507
### What changes were proposed in this pull request?
We make the tuple sketch build/agg (`UpdatableTupleSketchBuffer` in `tupleSketchState.scala`) also return the trimmed version: `serialize()` and `eval()` call `UpdatableSketch.trim()` before `compact`. `trim()` is public API of datasketches-java, so no library change is necessary.
The trimmed sketch keeps the k = 2^lgNomEntries smallest hashes, the same entries that the union keeps. As a result, every plan returns the same sketch.
### Why are the changes needed?
Bug description, all regarding `tuple_sketch_agg`:
- Spark GROUP BY returns a different answer depending on if it ran on one task (Complete agg mode) or in multiple tasks. When running on a single task it will return an untrimmed answer (taking 2x more memory) when compared to running on multiple tasks.
- Another side effect of this bug: when running in partial mode, the output for the leaf tasks will take 2x memory it would usually take.
- Another side effect of this bug: `tuple_union_agg`, when given the same inputs as `tuple_sketch_agg`, will produce a different estimate.
- Another side effect of this bug: outputs of `tuple_sketch_agg` window aggregates take 2x more memory (they are always untrimmed), which is non trivial as window aggregates produce approximately as many outputs as there are inputs (except for unbounded frames).
- Another side effect of this bug: when running on one task (Complete mode), the later entries in the sketch have a disproportionately large effect compared to earlier entries (this is not true when you trim), which can be bad if the later values are from a different distribution (for example they have much larger summaries). This breaks the order insensitivity of tuple agg.
Why this bug happens: in `tupleSketchState.scala`, for `tuple_sketch_agg` we sometimes use union instead (for merging results, eg. in Final aggregate mode). Union (`UnionTupleAggregationBuffer`) trims and tuple sketch build/agg (`UpdatableTupleSketchBuffer`) does not. A Complete-mode aggregate (for example with `spark.sql.execution.bypassPartialAggregation`, or with `spark.sql.execution.combineAdjacentAggregation` when no shuffle is needed) and a window never go through the union.
### Does this PR introduce _any_ user-facing change?
Yes. Perhaps breaking changes to users: this modifies the output of preexisting Spark GROUP BY (in Complete mode) and Spark window. The outputs would sometimes be 2x smaller, and would produce a different estimate (the multi-task estimate). In released versions, this affects the window in 4.2.0. The Complete-mode GROUP BY needs `combineAdjacentAggregation` or `bypassPartialAggregation`, which were added in 4.3.0.
Repro: put 1 or 4 in place of `N`. Before this PR, 1 gives the untrimmed result (28 and 24 entries) and 4 gives the trimmed result (16 and 16 entries, k = 16). With this PR, both give 16 and 16. This repro does not apply to 4.2.0.
```sql
SELECT g, (length(s) - 24) DIV 17 AS entries
FROM (SELECT id % 2 AS g, tuple_sketch_agg_double(id, 1.0D, 4) AS s
FROM range(0, 1000, 1, N)
GROUP BY id % 2);
```
### How was this patch tested?
All new tests use more than 2 * k distinct keys per group, so every sketch is in estimation mode. Without the fix, all 16 new tests fail.
**Unit tests** (`DataFrameAggregateSuite`, 16 new tests):
- Complete mode (2 flavors x 2 lg x 2 confs = 8 tests): GROUP BY `tuple_sketch_agg_<flavor>(key, summ, lg)` with `bypassPartialAggregation` or `combineAdjacentAggregation` on, against both off. Flavors: double, integer. lg: 5, 10. Each test asserts the aggregate modes (Complete against Partial + Final), k entries in every sketch, and the same theta and entries.
- Window equals GROUP BY (2 flavors x 4 summary modes = 8 tests): `tuple_sketch_agg_<flavor>(keyv % 100, summ, 5, mode) OVER (PARTITION BY pk)` against the same aggregate in a GROUP BY. Modes: sum, min, max, alwaysone. Every window row has the GROUP BY (estimate, theta, summary), with theta < 1.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac (Claude Code, Claude Opus 5.5)
This contribution is my original work, and I license the work to the project under the project's open source license.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dtenedor (via GitHub) via reviews" <re...@spark.apache.org>.
dtenedor commented on PR #59507:
URL: https://github.com/apache/spark/pull/59507#issuecomment-6589137083
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/904542dc061ef6c90ae51b9e1ba0dee71cd4139d
- merged into branch-4.x https://github.com/apache/spark/commit/a182242197f7c3466caa49acce78065fda9e6bc8
- merged into branch-4.3 https://github.com/apache/spark/commit/af442baaa053d47dc119ee616728f66849f26975
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dongjoon-hyun (via GitHub) via reviews" <re...@spark.apache.org>.
dongjoon-hyun commented on PR #59507:
URL: https://github.com/apache/spark/pull/59507#issuecomment-6637404404
Oh, this seems to break `branch-4.x` too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dtenedor (via GitHub) via reviews" <re...@spark.apache.org>.
dtenedor closed pull request #59507: [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan
URL: https://github.com/apache/spark/pull/59507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dongjoon-hyun (via GitHub) via reviews" <re...@spark.apache.org>.
dongjoon-hyun commented on PR #59507:
URL: https://github.com/apache/spark/pull/59507#issuecomment-6637591879
I fixed it vi at the following commit.
- https://github.com/apache/spark/commit/b8e4d33f889abedba9918f833d18909a54ec0d1a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dongjoon-hyun (via GitHub) via reviews" <re...@spark.apache.org>.
dongjoon-hyun commented on PR #59507:
URL: https://github.com/apache/spark/pull/59507#issuecomment-6623269165
I made a follow-up PR.
- https://github.com/apache/spark/pull/60080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dongjoon-hyun (via GitHub) via reviews" <re...@spark.apache.org>.
dongjoon-hyun commented on PR #59507:
URL: https://github.com/apache/spark/pull/59507#issuecomment-6623097396
Hi, @stojkomilos and @dtenedor .
Unfortunately, this broke `branch-4.3` compilation. Could you take a look at the failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dtenedor (via GitHub) via reviews" <re...@spark.apache.org>.
dtenedor commented on PR #59507:
URL: https://github.com/apache/spark/pull/59507#issuecomment-6589013391
LGTM. We should fix this behavior now according to the reasons described in the PR description. We'd want to merge this to the master branch, 4.x and 4.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60240][SQL] Trim tuple_sketch_agg results to nominal k entries in every plan [spark]
Posted by "dtenedor (via GitHub) via reviews" <re...@spark.apache.org>.
dtenedor commented on PR #59507:
URL: https://github.com/apache/spark/pull/59507#issuecomment-6692070878
Thank you @dongjoon-hyun for the fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org