You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jiwen624 (via GitHub)" <gi...@apache.org> on 2026/07/17 06:15:46 UTC
[PR] [SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion [spark]
jiwen624 opened a new pull request, #57327:
URL: https://github.com/apache/spark/pull/57327
### What changes were proposed in this pull request?
`approx_top_k_estimate` picked its sketch deserialization SerDe from the static item type in the sketch-state struct (field 2). A `UNION ALL` of two sketch columns with coercible item types (e.g. INT + BIGINT) widens that static type while the serialized bytes and the per-row DDL (field 3) keep the original type. Estimation then decoded the bytes with the wrong SerDe.
This PR makes `estimate` deserialize using the true item type recorded per-row in the state DDL (recovering collation from the static type, exactly as `combine` already does), then widens each decoded item to the declared output type via a lossless cast so results still match the statically-declared schema.
### Why are the changes needed?
A query like:
```sql
SELECT approx_top_k_estimate(approx_top_k_combine(sketch, 5))
FROM (SELECT approx_top_k_accumulate(c) AS sketch, 'int' AS tag
FROM VALUES (1), (1), (2) AS t(c)
UNION ALL
SELECT approx_top_k_accumulate(c) AS sketch, 'long' AS tag
FROM VALUES (CAST(9 AS BIGINT)) AS t(c))
WHERE tag = 'int';
```
fails at runtime with SketchesArgumentException: Bounds Violation, e.g.,
```Java
26/07/16 22:29:33 ERROR Executor: Exception in task 0.0 in stage 2.0 (TID 3)
org.apache.datasketches.common.SketchesArgumentException: Bounds Violation: reqOffset: 0, reqLength: 16, (reqOff + reqLen): 16, allocSize: 10
at org.apache.datasketches.common.Util.checkBounds(Util.java:659)
```
Or, depending on the widening it can also throw ClassCastException or, when byte sizes
happen to match (e.g. BIGINT→DOUBLE), silently return wrong results.
### Does this PR introduce _any_ user-facing change?
Yes. A query that may crash or return wrong results now gets fixed.
### How was this patch tested?
Added UT.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57327:
URL: https://github.com/apache/spark/pull/57327#issuecomment-5059562235
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/490f2bbf1d783c9fd4531d0dcbbedbcfc795b3a3
- merged into branch-4.x https://github.com/apache/spark/commit/350fa45aa4db9757a6304b499b96db8d0022814e
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57327:
URL: https://github.com/apache/spark/pull/57327#issuecomment-5048051350
Please resolve conflicts. Also cc @cloud-fan @yhuang-db
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion [spark]
Posted by "jiwen624 (via GitHub)" <gi...@apache.org>.
jiwen624 commented on PR #57327:
URL: https://github.com/apache/spark/pull/57327#issuecomment-5041871768
@uros-b @cloud-fan this is the fix of another issue of approx_top_k
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #57327: [SPARK-58164][SQL] approx_top_k_estimate crashes or returns wrong results on sketches whose item type was widened by type coercion
URL: https://github.com/apache/spark/pull/57327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org