You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Daniel Tenedorio (Jira)" <ji...@apache.org> on 2025/11/03 22:04:00 UTC
[jira] [Comment Edited] (SPARK-53991) Add support for KLL quantiles functions based on DataSketches

    [ https://issues.apache.org/jira/browse/SPARK-53991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032269#comment-18032269 ] 

Daniel Tenedorio edited comment on SPARK-53991 at 11/3/25 10:03 PM:
--------------------------------------------------------------------

We can use the following function names. For each category, we can support long integer, single-precision floating-point, and double-precision floating-point variants (it is necessary to specify which one we are using since the representation of the sketch buffers is different for different input value data types).

 

Aggregate functions to consume input values and return a sketch buffer:

{{kll_sketch_agg_bigint(col)}}

{{kll_sketch_agg_float(col)}}

{{kll_sketch_agg_double(col)}}

 

Scalar functions to merge two sketch buffers together into another sketch buffer:

{{kll_sketch_merge_bigint(sketch1, sketch2)}}

{{kll_sketch_merge_float(sketch1, sketch2)}}

{{kll_sketch_merge_double(sketch1, sketch2)}}

 

Scalar functions to extract a single value from the quantiles sketch representing the desired quantile given the input rank (for example, "float median = sketch.getQuantile(0.5)"). We can also implement the functions in this category to also support accepting an array of input ranks and return an array of result quantiles.

{{kll_sketch_get_quantile_bigint(sketch, rank)}}

{{kll_sketch_get_quantile_float(sketch, rank)}}

{{kll_sketch_get_quantile_double(sketch, rank)}}

 

Scalar functions to extract a single value from the quantiles sketch representing the desired rank given the input quantile (for example, "double rankOf1000 = sketch.getRank(1000)"). We can also implement the functions in this category to also support accepting an array of input quantiles and return an array of result ranks.

{{kll_sketch_get_rank_bigint(sketch, quantile)}}

{{kll_sketch_get_rank_float(sketch, quantile)}}

{{kll_sketch_get_rank_double(sketch, quantile)}}


Scalar functions to count the number of items collected in the sketch so far.

{{kll_sketch_get_n_bigint(sketch)}}

{{kll_sketch_get_n_float(sketch)}}

{{kll_sketch_get_n_double(sketch)}}


Optional, scalar functions to return a string representation of a sketch buffer:

{{kll_sketch_to_string_bigint(sketch)}}

{{kll_sketch_to_string_float(sketch)}}

{{kll_sketch_to_string_double(sketch)}}



was (Author: JIRAUSER285772):
We can use the following function names. For each category, we can support long integer, single-precision floating-point, and double-precision floating-point variants (it is necessary to specify which one we are using since the representation of the sketch buffers is different for different input value data types).

 

Aggregate functions to consume input values and return a sketch buffer:

{{kll_sketch_agg_bigint(col)}}

{{kll_sketch_agg_float(col)}}

{{kll_sketch_agg_double(col)}}

 

Scalar functions to merge two sketch buffers together into another sketch buffer:

{{kll_sketch_merge_bigint(sketch1, sketch2)}}

{{kll_sketch_merge_float(sketch1, sketch2)}}

{{kll_sketch_merge_double(sketch1, sketch2)}}

 

Scalar functions to extract a single value from the quantiles sketch representing the desired quantile given the input rank (for example, "float median = sketch.getQuantile(0.5)"). We can also implement the functions in this category to also support accepting an array of input ranks and return an array of result quantiles.

{{kll_sketch_get_quantile_bigint(sketch, rank)}}

{{kll_sketch_get_quantile_float(sketch, rank)}}

{{kll_sketch_get_quantile_double(sketch, rank)}}

 

Scalar functions to extract a single value from the quantiles sketch representing the desired rank given the input quantile (for example, "double rankOf1000 = sketch.getRank(1000)"). We can also implement the functions in this category to also support accepting an array of input quantiles and return an array of result ranks.

{{kll_sketch_get_rank_bigint(sketch, quantile)}}

{{kll_sketch_get_rank_float(sketch, quantile)}}

{{kll_sketch_get_rank_double(sketch, quantile)}}

Scalar functions to count the number of items collected in the sketch so far.

{{kll_sketch_get_n_bigint(sketch)}}

{{kll_sketch_get_n_float(sketch)}}

{{kll_sketch_get_n_double(sketch)}}

Optional, scalar functions to return a string representation of a sketch buffer:

{{kll_sketch_to_string_bigint(sketch)}}

{{kll_sketch_to_string_float(sketch)}}

{{kll_sketch_to_string_double(sketch)}}


> Add support for KLL quantiles functions based on DataSketches
> -------------------------------------------------------------
>
>                 Key: SPARK-53991
>                 URL: https://issues.apache.org/jira/browse/SPARK-53991
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>    Affects Versions: 4.1.0
>            Reporter: Daniel Tenedorio
>            Assignee: Daniel Tenedorio
>            Priority: Major
>              Labels: pull-request-available
>
> Documentation reference: [https://datasketches.apache.org/docs/KLL/KLLSketch.html].
> DataSketches code API reference: [https://apache.github.io/datasketches-java/6.1.0/org/apache/datasketches/kll/KllLongsSketch.html] 
> Reference PR for recently adding Theta sketches: [https://github.com/apache/spark/pull/51298]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org