You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/07/21 07:57:00 UTC
(spark) branch master updated: [SPARK-58234][PS] Use native functions for NumPy inverse hyperbolic ufuncs
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 776fe8d7c95d [SPARK-58234][PS] Use native functions for NumPy inverse hyperbolic ufuncs
776fe8d7c95d is described below
commit 776fe8d7c95de14598ad2c277feffcc571cc6c0d
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Tue Jul 21 15:56:41 2026 +0800
[SPARK-58234][PS] Use native functions for NumPy inverse hyperbolic ufuncs
### What changes were proposed in this pull request?
Replace the scalar pandas UDF mappings for NumPy `arccosh`, `arcsinh`, and `arctanh` on pandas-on-Spark objects with the native Spark SQL functions `acosh`, `asinh`, and `atanh`.
Add end-to-end pandas-on-Spark coverage that compares each NumPy ufunc result with pandas.
### Why are the changes needed?
These mappings originated in Koalas when it supported PySpark 2.4, before the native inverse-hyperbolic Spark SQL functions were available. Current Spark provides the native functions, including Spark Connect support, so this removes the Python worker boundary while preserving NumPy-compatible results.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added end-to-end pandas-on-Spark coverage for `arccosh`, `arcsinh`, and `arctanh`.
- Ran `git diff --check`.
- Runtime tests were not run locally.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57397 from zhengruifeng/pandas-native-inverse-hyperbolic-dev2.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/pandas/numpy_compat.py | 6 +++---
python/pyspark/pandas/tests/test_numpy_compat.py | 13 +++++++++++++
2 files changed, 16 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py b/python/pyspark/pandas/numpy_compat.py
index 913089ca56c4..e43dd8a1094e 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -27,11 +27,11 @@ unary_np_spark_mappings = {
"abs": F.abs,
"absolute": F.abs,
"arccos": F.acos,
- "arccosh": pandas_udf(lambda s: np.arccosh(s), DoubleType()), # type: ignore[call-overload]
+ "arccosh": F.acosh,
"arcsin": F.asin,
- "arcsinh": pandas_udf(lambda s: np.arcsinh(s), DoubleType()), # type: ignore[call-overload]
+ "arcsinh": F.asinh,
"arctan": F.atan,
- "arctanh": pandas_udf(lambda s: np.arctanh(s), DoubleType()), # type: ignore[call-overload]
+ "arctanh": F.atanh,
"bitwise_not": F.bitwiseNOT,
"cbrt": F.cbrt,
"ceil": F.ceil,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index 02386ccbbecf..d04c2c135a5d 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -85,6 +85,19 @@ class NumPyCompatTestsMixin:
with self.assertRaisesRegex(ValueError, "cannot join with no overlapping index names"):
np.left_shift(psdf1, psdf2)
+ def test_np_inverse_hyperbolic_series_uses_native_functions(self):
+ for np_func, values in (
+ (np.arccosh, [1.0, 2.0, 4.0]),
+ (np.arcsinh, [-2.0, 0.0, 2.0]),
+ (np.arctanh, [-0.5, 0.0, 0.5]),
+ ):
+ with self.subTest(name=np_func.__name__):
+ pdf = pd.DataFrame({"a": values})
+ psdf = ps.from_pandas(pdf)
+ result = np_func(psdf.a)
+
+ self.assert_eq(result, np_func(pdf.a), almost=True)
+
def test_np_spark_compat_series(self):
from pyspark.pandas.numpy_compat import unary_np_spark_mappings, binary_np_spark_mappings
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org