You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dtenedor (via GitHub)" <gi...@apache.org> on 2025/11/05 19:55:58 UTC
[PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
dtenedor opened a new pull request, #52900:
URL: https://github.com/apache/spark/pull/52900
### What changes were proposed in this pull request?
This PR adds DataFrame API support for the KLL quantile sketch functions that were previously added to Spark SQL in https://github.com/apache/spark/pull/52800. This lets users leverage KLL sketches through both Scala and Python DataFrame APIs in addition to the existing SQL interface.
**Key additions:**
1. **Scala DataFrame API** (`sql/api/src/main/scala/org/apache/spark/sql/functions.scala`):
- 18 new functions covering aggregate, merge, quantile, and rank operations
- Multiple overloads for each function supporting:
- `Column` parameters for computed values
- `String` parameters for column names
- `Int` parameters for literal k values
- Optional k parameters with sensible defaults
- Functions for all three data type variants: bigint, float, double
2. **Python DataFrame API** (`python/pyspark/sql/functions/builtin.py`):
- 18 corresponding Python functions with:
- Comprehensive docstrings with usage examples
- Proper type hints (`ColumnOrName`, `Optional[Union[int, Column]]`)
- Support for both column objects and column name strings
- Added to PySpark documentation reference
3. **Python Spark Connect Support** (`python/pyspark/sql/connect/functions/builtin.py`):
- Full compatibility with Spark Connect architecture
- All 18 functions properly registered
### Why are the changes needed?
While the SQL API for KLL sketches was previously added, DataFrame API support is essential for:
- **Programmatic workflows**: Users building Spark applications in Scala/Python need native API access
- **Type safety**: DataFrame API provides compile-time type checking in Scala
- **Composability**: Easier to chain operations and integrate with existing DataFrame pipelines
- **Consistency**: All other sketch functions (HLL, Theta) provide DataFrame APIs
- **IDE support**: Better autocomplete and inline documentation for developers
Without DataFrame API support, users would be forced to use SQL expressions via `expr()` or `selectExpr()`, which is less ergonomic and type-safe.
### Does this PR introduce any user-facing change?
Yes, this PR adds DataFrame API support for the 18 KLL sketch functions:
**Scala DataFrame API Example:**
```scala
import org.apache.spark.sql.functions._
// Create sketch with default k
val df = Seq(1, 2, 3, 4, 5).toDF("value")
val sketch = df.agg(kll_sketch_agg_bigint($"value"))
// Create sketch with custom k value
val sketch2 = df.agg(kll_sketch_agg_bigint("value", 400))
// Get median (0.5 quantile)
val sketchDf = df.agg(kll_sketch_agg_bigint($"value").alias("sketch"))
val median = sketchDf.select(kll_sketch_get_quantile_bigint($"sketch", lit(0.5)))
// Get multiple quantiles
val quantiles = sketchDf.select(
kll_sketch_get_quantile_bigint($"sketch", array(lit(0.25), lit(0.5), lit(0.75)))
)
// Merge sketches
val merged = sketchDf.select(
kll_sketch_merge_bigint($"sketch", $"sketch").alias("merged")
)
// Get count of items
val count = sketchDf.select(kll_sketch_get_n_bigint($"sketch"))
```
**Python DataFrame API Example:**
```python
from pyspark.sql import functions as sf
# Create sketch with default k
df = spark.createDataFrame([1, 2, 3, 4, 5], "INT")
sketch = df.agg(sf.kll_sketch_agg_bigint("value"))
# Create sketch with custom k value
sketch2 = df.agg(sf.kll_sketch_agg_bigint("value", 400))
# Get median (0.5 quantile)
sketch_df = df.agg(sf.kll_sketch_agg_bigint("value").alias("sketch"))
median = sketch_df.select(sf.kll_sketch_get_quantile_bigint("sketch", sf.lit(0.5)))
# Get multiple quantiles
quantiles = sketch_df.select(
sf.kll_sketch_get_quantile_bigint("sketch", sf.array(sf.lit(0.25), sf.lit(0.5), sf.lit(0.75)))
)
# Merge sketches
merged = sketch_df.select(
sf.kll_sketch_merge_bigint("sketch", "sketch").alias("merged")
)
# Get count of items
count = sketch_df.select(sf.kll_sketch_get_n_bigint("sketch"))
```
### How was this patch tested?
1. **Scala Unit Tests** (`DataFrameAggregateSuite`):
- `kll_sketch_agg_{bigint,float,double}` with default and explicit k values
- `kll_sketch_to_string` functions for all data types
- `kll_sketch_get_n` functions for all data types
- `kll_sketch_merge` operations
- `kll_sketch_get_quantile` with single rank and array of ranks
- `kll_sketch_get_rank` operations
- Null value handling tests
- **Result**: All 11 tests pass successfully
2. **Python Unit Tests** (`test_functions.py`):
- Comprehensive tests mirroring Scala tests
- Tests for Column object and string column name overloads
- Tests for optional k parameter
- Array input tests for quantile/rank functions
- Null handling validation
- Type checking (bytes/bytearray for sketches, str for to_string, int/float for values)
### Was this patch authored or co-authored using generative AI tooling?
Yes, IDE assistance used `claude-4.5-sonnet` with manual validation and integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on code in PR #52900:
URL: https://github.com/apache/spark/pull/52900#discussion_r2496016931
##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -4529,6 +4529,168 @@ def theta_intersection_agg(
theta_intersection_agg.__doc__ = pysparkfuncs.theta_intersection_agg.__doc__
+def kll_sketch_agg_bigint(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_bigint"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_bigint.__doc__ = pysparkfuncs.kll_sketch_agg_bigint.__doc__
+
+
+def kll_sketch_agg_float(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_float"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_float.__doc__ = pysparkfuncs.kll_sketch_agg_float.__doc__
+
+
+def kll_sketch_agg_double(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_double"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_double.__doc__ = pysparkfuncs.kll_sketch_agg_double.__doc__
+
+
+def kll_sketch_to_string_bigint(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_bigint"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_bigint.__doc__ = pysparkfuncs.kll_sketch_to_string_bigint.__doc__
+
+
+def kll_sketch_to_string_float(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_float"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_float.__doc__ = pysparkfuncs.kll_sketch_to_string_float.__doc__
+
+
+def kll_sketch_to_string_double(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_double"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_double.__doc__ = pysparkfuncs.kll_sketch_to_string_double.__doc__
+
+
+def kll_sketch_get_n_bigint(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_bigint"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_bigint.__doc__ = pysparkfuncs.kll_sketch_get_n_bigint.__doc__
+
+
+def kll_sketch_get_n_float(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_float"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_float.__doc__ = pysparkfuncs.kll_sketch_get_n_float.__doc__
+
+
+def kll_sketch_get_n_double(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_double"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_double.__doc__ = pysparkfuncs.kll_sketch_get_n_double.__doc__
+
+
+def kll_sketch_merge_bigint(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_bigint"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_bigint.__doc__ = pysparkfuncs.kll_sketch_merge_bigint.__doc__
+
+
+def kll_sketch_merge_float(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_float"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_float.__doc__ = pysparkfuncs.kll_sketch_merge_float.__doc__
+
+
+def kll_sketch_merge_double(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_double"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_double.__doc__ = pysparkfuncs.kll_sketch_merge_double.__doc__
+
+
+def kll_sketch_get_quantile_bigint(sketch: "ColumnOrName", rank: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_quantile_bigint"
+ return _invoke_function_over_columns(fn, sketch, rank)
+
+
+kll_sketch_get_quantile_bigint.__doc__ = pysparkfuncs.kll_sketch_get_quantile_bigint.__doc__
+
+
+def kll_sketch_get_quantile_float(sketch: "ColumnOrName", rank: "ColumnOrName") -> Column:
Review Comment:
noticed that in the line below, we don't check if the ranks column type is not supported, not sure what would happen in that case. https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/kllExpressions.scala#L488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52900:
URL: https://github.com/apache/spark/pull/52900#discussion_r2496045863
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -1172,7 +1172,7 @@ object functions {
* column via a Datasketches Intersection instance.
*
* @group agg_funcs
- * @since 4.1.0
+ * @since 4.2.0
Review Comment:
My mistake on this version change, it looks like 4.1.0 is the correct version after all. Reverted that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3500221290
Thank you, @cloud-fan !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3519892616
@dongjoon-hyun my mistake on the review protocols, I did not know that part as a new committer; I will get another committer(s) review for my own future work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3517332455
I've backported both https://github.com/apache/spark/pull/52800 and this PR to 4.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on code in PR #52900:
URL: https://github.com/apache/spark/pull/52900#discussion_r2496119351
##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -4529,6 +4529,168 @@ def theta_intersection_agg(
theta_intersection_agg.__doc__ = pysparkfuncs.theta_intersection_agg.__doc__
+def kll_sketch_agg_bigint(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_bigint"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_bigint.__doc__ = pysparkfuncs.kll_sketch_agg_bigint.__doc__
+
+
+def kll_sketch_agg_float(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_float"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_float.__doc__ = pysparkfuncs.kll_sketch_agg_float.__doc__
+
+
+def kll_sketch_agg_double(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_double"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_double.__doc__ = pysparkfuncs.kll_sketch_agg_double.__doc__
+
+
+def kll_sketch_to_string_bigint(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_bigint"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_bigint.__doc__ = pysparkfuncs.kll_sketch_to_string_bigint.__doc__
+
+
+def kll_sketch_to_string_float(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_float"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_float.__doc__ = pysparkfuncs.kll_sketch_to_string_float.__doc__
+
+
+def kll_sketch_to_string_double(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_double"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_double.__doc__ = pysparkfuncs.kll_sketch_to_string_double.__doc__
+
+
+def kll_sketch_get_n_bigint(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_bigint"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_bigint.__doc__ = pysparkfuncs.kll_sketch_get_n_bigint.__doc__
+
+
+def kll_sketch_get_n_float(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_float"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_float.__doc__ = pysparkfuncs.kll_sketch_get_n_float.__doc__
+
+
+def kll_sketch_get_n_double(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_double"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_double.__doc__ = pysparkfuncs.kll_sketch_get_n_double.__doc__
+
+
+def kll_sketch_merge_bigint(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_bigint"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_bigint.__doc__ = pysparkfuncs.kll_sketch_merge_bigint.__doc__
+
+
+def kll_sketch_merge_float(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_float"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_float.__doc__ = pysparkfuncs.kll_sketch_merge_float.__doc__
+
+
+def kll_sketch_merge_double(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_double"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_double.__doc__ = pysparkfuncs.kll_sketch_merge_double.__doc__
+
+
+def kll_sketch_get_quantile_bigint(sketch: "ColumnOrName", rank: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_quantile_bigint"
+ return _invoke_function_over_columns(fn, sketch, rank)
+
+
+kll_sketch_get_quantile_bigint.__doc__ = pysparkfuncs.kll_sketch_get_quantile_bigint.__doc__
+
+
+def kll_sketch_get_quantile_float(sketch: "ColumnOrName", rank: "ColumnOrName") -> Column:
Review Comment:
Ah then the checkInputDataTypes function probably does the check when the expression is resolved. I missed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3493077578
cc @cboumalh here is the DataFrame API support for the new functions added in https://github.com/apache/spark/pull/52800, do you want to take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3499701437
Thanks for your review!
I just did another manual review as well.
Everything looks good, merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3499776188
In addition, if this is not targeting for Apache Spark 4.1.0 technically, please update the wrong `since` tags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor closed pull request #52900: [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions
URL: https://github.com/apache/spark/pull/52900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3519881446
Thank you @cloud-fan for the review + backporting 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "cboumalh (via GitHub)" <gi...@apache.org>.
cboumalh commented on code in PR #52900:
URL: https://github.com/apache/spark/pull/52900#discussion_r2495958692
##########
sql/api/src/main/scala/org/apache/spark/sql/functions.scala:
##########
@@ -1172,7 +1172,7 @@ object functions {
* column via a Datasketches Intersection instance.
*
* @group agg_funcs
- * @since 4.1.0
+ * @since 4.2.0
Review Comment:
Do we want to add something in the PR description about these versioning changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3499713156
Hi, @dtenedor . I don't see any committers' approval yet.
> Thanks for your review! I just did another manual review as well. Everything looks good, merging to master.
<img width="347" height="106" alt="Screenshot 2025-11-06 at 14 51 34" src="https://github.com/user-attachments/assets/0b9bb5d2-1061-4f72-91c4-b09a7d41d1a3" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #52900:
URL: https://github.com/apache/spark/pull/52900#issuecomment-3499965200
The change LGTM, it's just adding dataframe functions to match SQL.
Given the related SQL functions are in 4.1, I think it makes sense to backport this to 4.1 as well. @dtenedor what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions [spark]
Posted by "dtenedor (via GitHub)" <gi...@apache.org>.
dtenedor commented on code in PR #52900:
URL: https://github.com/apache/spark/pull/52900#discussion_r2496087502
##########
python/pyspark/sql/connect/functions/builtin.py:
##########
@@ -4529,6 +4529,168 @@ def theta_intersection_agg(
theta_intersection_agg.__doc__ = pysparkfuncs.theta_intersection_agg.__doc__
+def kll_sketch_agg_bigint(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_bigint"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_bigint.__doc__ = pysparkfuncs.kll_sketch_agg_bigint.__doc__
+
+
+def kll_sketch_agg_float(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_float"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_float.__doc__ = pysparkfuncs.kll_sketch_agg_float.__doc__
+
+
+def kll_sketch_agg_double(
+ col: "ColumnOrName",
+ k: Optional[Union[int, Column]] = None,
+) -> Column:
+ fn = "kll_sketch_agg_double"
+ if k is None:
+ return _invoke_function_over_columns(fn, col)
+ else:
+ return _invoke_function_over_columns(fn, col, lit(k))
+
+
+kll_sketch_agg_double.__doc__ = pysparkfuncs.kll_sketch_agg_double.__doc__
+
+
+def kll_sketch_to_string_bigint(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_bigint"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_bigint.__doc__ = pysparkfuncs.kll_sketch_to_string_bigint.__doc__
+
+
+def kll_sketch_to_string_float(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_float"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_float.__doc__ = pysparkfuncs.kll_sketch_to_string_float.__doc__
+
+
+def kll_sketch_to_string_double(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_to_string_double"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_to_string_double.__doc__ = pysparkfuncs.kll_sketch_to_string_double.__doc__
+
+
+def kll_sketch_get_n_bigint(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_bigint"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_bigint.__doc__ = pysparkfuncs.kll_sketch_get_n_bigint.__doc__
+
+
+def kll_sketch_get_n_float(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_float"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_float.__doc__ = pysparkfuncs.kll_sketch_get_n_float.__doc__
+
+
+def kll_sketch_get_n_double(col: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_n_double"
+ return _invoke_function_over_columns(fn, col)
+
+
+kll_sketch_get_n_double.__doc__ = pysparkfuncs.kll_sketch_get_n_double.__doc__
+
+
+def kll_sketch_merge_bigint(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_bigint"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_bigint.__doc__ = pysparkfuncs.kll_sketch_merge_bigint.__doc__
+
+
+def kll_sketch_merge_float(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_float"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_float.__doc__ = pysparkfuncs.kll_sketch_merge_float.__doc__
+
+
+def kll_sketch_merge_double(left: "ColumnOrName", right: "ColumnOrName") -> Column:
+ fn = "kll_sketch_merge_double"
+ return _invoke_function_over_columns(fn, left, right)
+
+
+kll_sketch_merge_double.__doc__ = pysparkfuncs.kll_sketch_merge_double.__doc__
+
+
+def kll_sketch_get_quantile_bigint(sketch: "ColumnOrName", rank: "ColumnOrName") -> Column:
+ fn = "kll_sketch_get_quantile_bigint"
+ return _invoke_function_over_columns(fn, sketch, rank)
+
+
+kll_sketch_get_quantile_bigint.__doc__ = pysparkfuncs.kll_sketch_get_quantile_bigint.__doc__
+
+
+def kll_sketch_get_quantile_float(sketch: "ColumnOrName", rank: "ColumnOrName") -> Column:
Review Comment:
Looking at this, I think the class mixing in `ImplicitCastInputTypes` should make sure that both arguments have either the same types or are coercible to the required types. I'll add a couple negative tests just to make sure.
```
abstract class KllSketchGetQuantileBase
extends BinaryExpression
with CodegenFallback
with ImplicitCastInputTypes {
...
override def inputTypes: Seq[AbstractDataType] =
Seq(
BinaryType,
TypeCollection(
DoubleType,
ArrayType(DoubleType, containsNull = false)))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org