You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2024/11/18 11:19:10 UTC
(spark) branch master updated: [MINOR][PYTHON][DOCS] Fix the type hint of `histogram_numeric`
This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 05750def2da6 [MINOR][PYTHON][DOCS] Fix the type hint of `histogram_numeric`
05750def2da6 is described below
commit 05750def2da6fe16fc0cf8a3eee79bb2056df979
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Nov 18 19:18:46 2024 +0800
[MINOR][PYTHON][DOCS] Fix the type hint of `histogram_numeric`
### What changes were proposed in this pull request?
Fix the type hint of `histogram_numeric`
### Why are the changes needed?
argument `nBins` cannot be a column name:
```
In [11]: spark.range(100).withColumn("nb", sf.lit(5)).select(sf.histogram_numeric('id', "nb")).show(truncate=False)
...
AnalysisException: [DATATYPE_MISMATCH.NON_FOLDABLE_INPUT] Cannot resolve "histogram_numeric(id, nb)" due to data type mismatch: the input `nb` should be a foldable "INT" expression; however, got "nb". SQLSTATE: 42K09;
'Aggregate [unresolvedalias(histogram_numeric(id#323L, nb#324, 0, 0))]
+- Project [id#323L, 5 AS nb#324]
+- Range (0, 100, step=1, splits=Some(12))
```
### Does this PR introduce _any_ user-facing change?
doc-only
### How was this patch tested?
updated doctest
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #48875 from zhengruifeng/fix_histogram_numeric.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/sql/connect/functions/builtin.py | 2 +-
python/pyspark/sql/functions/builtin.py | 25 +++++++++++--------------
2 files changed, 12 insertions(+), 15 deletions(-)
diff --git a/python/pyspark/sql/connect/functions/builtin.py b/python/pyspark/sql/connect/functions/builtin.py
index fa1a81ab04eb..f52cdffb84b7 100644
--- a/python/pyspark/sql/connect/functions/builtin.py
+++ b/python/pyspark/sql/connect/functions/builtin.py
@@ -1552,7 +1552,7 @@ def count_if(col: "ColumnOrName") -> Column:
count_if.__doc__ = pysparkfuncs.count_if.__doc__
-def histogram_numeric(col: "ColumnOrName", nBins: "ColumnOrName") -> Column:
+def histogram_numeric(col: "ColumnOrName", nBins: Column) -> Column:
return _invoke_function_over_columns("histogram_numeric", col, nBins)
diff --git a/python/pyspark/sql/functions/builtin.py b/python/pyspark/sql/functions/builtin.py
index 842fdf7ae423..1beaa4807ef3 100644
--- a/python/pyspark/sql/functions/builtin.py
+++ b/python/pyspark/sql/functions/builtin.py
@@ -8557,7 +8557,7 @@ def count_if(col: "ColumnOrName") -> Column:
@_try_remote_functions
-def histogram_numeric(col: "ColumnOrName", nBins: "ColumnOrName") -> Column:
+def histogram_numeric(col: "ColumnOrName", nBins: Column) -> Column:
"""Computes a histogram on numeric 'col' using nb bins.
The return value is an array of (x,y) pairs representing the centers of the
histogram's bins. As the value of 'nb' is increased, the histogram approximation
@@ -8573,9 +8573,9 @@ def histogram_numeric(col: "ColumnOrName", nBins: "ColumnOrName") -> Column:
Parameters
----------
- col : :class:`~pyspark.sql.Column` or str
+ col : :class:`~pyspark.sql.Column` or column name
target column to work on.
- nBins : :class:`~pyspark.sql.Column` or str
+ nBins : :class:`~pyspark.sql.Column`
number of Histogram columns.
Returns
@@ -8585,17 +8585,14 @@ def histogram_numeric(col: "ColumnOrName", nBins: "ColumnOrName") -> Column:
Examples
--------
- >>> df = spark.createDataFrame([("a", 1),
- ... ("a", 2),
- ... ("a", 3),
- ... ("b", 8),
- ... ("b", 2)], ["c1", "c2"])
- >>> df.select(histogram_numeric('c2', lit(5))).show()
- +------------------------+
- |histogram_numeric(c2, 5)|
- +------------------------+
- | [{1, 1.0}, {2, 1....|
- +------------------------+
+ >>> from pyspark.sql import functions as sf
+ >>> df = spark.range(100, numPartitions=1)
+ >>> df.select(sf.histogram_numeric('id', sf.lit(5))).show(truncate=False)
+ +-----------------------------------------------------------+
+ |histogram_numeric(id, 5) |
+ +-----------------------------------------------------------+
+ |[{11, 25.0}, {36, 24.0}, {59, 23.0}, {84, 25.0}, {98, 3.0}]|
+ +-----------------------------------------------------------+
"""
return _invoke_function_over_columns("histogram_numeric", col, nBins)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org