You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "durgaprasadml (via GitHub)" <gi...@apache.org> on 2026/06/04 20:11:00 UTC
[PR] refactor: replace misleading approx_top_k terminology with frequent items naming [spark]
durgaprasadml opened a new pull request, #56333:
URL: https://github.com/apache/spark/pull/56333
## Summary
Rename approx_top_k terminology to approx_frequent_items across the Spark codebase to better reflect the semantics of the underlying Apache DataSketches Frequent Items sketch implementation.
The current naming is misleading because the sketch does not provide strict top-k guarantees. Instead, it identifies frequent items / heavy hitters based on threshold-oriented probabilistic guarantees and may legitimately return fewer than k items (or even none).
This change aligns Spark terminology more closely with the Apache DataSketches documentation and expected sketch behavior.
## Changes
### Core implementation
- Renamed Catalyst aggregate and expression implementations
- Renamed aggregate registration references
- Updated SQL function registration entries
- Updated related error definitions and references
### APIs
- Updated Scala SQL APIs
- Updated PySpark APIs
- Updated Spark Connect function definitions
### Documentation
- Updated sql-ref-sketch-aggregates.md
- Replaced misleading "top k" terminology with "frequent items" terminology
- Clarified sketch semantics and behavior
### Tests
- Renamed and updated related test suites
- Regenerated SQL expression schema golden files
- Fixed test expectations referencing old SQL names
- Verified FrequentItemsSuite passes successfully
## Motivation
The underlying implementation uses Apache DataSketches Frequent Items sketches, which:
- identify heavy hitters / frequent items
- provide threshold-based probabilistic guarantees
- do not guarantee ranked top-k semantics
For example, depending on stream distribution and sketch size, the sketch may validly return:
- fewer than k items
- or no items at all
The previous naming created incorrect expectations for users and did not match the terminology used by Apache DataSketches itself.
## Validation
Validated with:
- FrequentItemsSuite
- SQL expression schema regeneration
- compilation and API updates
- updated documentation and registry references
## Related Issue
Fixes #56331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] refactor: replace misleading approx_top_k terminology with frequent items naming [spark]
Posted by "durgaprasadml (via GitHub)" <gi...@apache.org>.
durgaprasadml commented on PR #56333:
URL: https://github.com/apache/spark/pull/56333#issuecomment-4651135895
> Mind filing a JIRA ticket please?
Created JIRA ticket for tracking:
SPARK-57319 - Rename misleading approx_top_k terminology to approx_frequent_items
https://issues.apache.org/jira/browse/SPARK-57319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] refactor: replace misleading approx_top_k terminology with frequent items naming [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56333:
URL: https://github.com/apache/spark/pull/56333#issuecomment-5706663428
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] refactor: replace misleading approx_top_k terminology with frequent items naming [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #56333: refactor: replace misleading approx_top_k terminology with frequent items naming
URL: https://github.com/apache/spark/pull/56333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] refactor: replace misleading approx_top_k terminology with frequent items naming [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56333:
URL: https://github.com/apache/spark/pull/56333#issuecomment-4646027179
Mind filing a JIRA ticket please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] refactor: replace misleading approx_top_k terminology with frequent items naming [spark]
Posted by "durgaprasadml (via GitHub)" <gi...@apache.org>.
durgaprasadml commented on PR #56333:
URL: https://github.com/apache/spark/pull/56333#issuecomment-4646090070
> Mind filing a JIRA ticket please?
Thanks for the reminder. I’ve requested ASF JIRA access so I can create the corresponding SPARK issue and link it to this PR. I’ll update the PR title and description with the JIRA ID once the account request is approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org