You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/04 09:31:17 UTC
(spark) branch master updated: [SPARK-58533][PS] Use native Spark function for NumPy float_power
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ad9c5320c36e [SPARK-58533][PS] Use native Spark function for NumPy float_power
ad9c5320c36e is described below
commit ad9c5320c36e9526cea28392bebb6d837e5c3bee
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Tue Aug 4 17:31:05 2026 +0800
[SPARK-58533][PS] Use native Spark function for NumPy float_power
### What changes were proposed in this pull request?
This PR replaces the pandas UDF implementation of `np.float_power` in pandas API on Spark with native Spark `pow`, casting both operands to `double`. It also adds compatibility coverage for integral inputs and floating-point special values.
### Why are the changes needed?
Using a native Spark expression avoids pandas UDF and Arrow overhead while preserving NumPy `float_power` floating-point semantics.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added `test_np_float_power`, covering integral inputs, signed zero, infinities, and NaN.
- `ruff check python/pyspark/pandas/numpy_compat.py python/pyspark/pandas/tests/test_numpy_compat.py`
- `ruff format --check python/pyspark/pandas/numpy_compat.py python/pyspark/pandas/tests/test_numpy_compat.py`
- `python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat` (`test_np_float_power` passed; the module has two unrelated existing reciprocal pandas-UDF failures due to a local Python/JVM class-signature mismatch.)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex GPT-5
Closes #57734 from zhengruifeng/pandas-native-float-power-dev2.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
python/pyspark/pandas/numpy_compat.py | 4 +---
python/pyspark/pandas/tests/test_numpy_compat.py | 17 +++++++++++++++++
2 files changed, 18 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py b/python/pyspark/pandas/numpy_compat.py
index 6bdb5e97841f..f333594fd08d 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -107,9 +107,7 @@ binary_np_spark_mappings = {
"copysign": pandas_udf( # type: ignore[call-overload]
lambda s1, s2: np.copysign(s1, s2), DoubleType()
),
- "float_power": pandas_udf( # type: ignore[call-overload]
- lambda s1, s2: np.float_power(s1, s2), DoubleType()
- ),
+ "float_power": lambda c1, c2: F.pow(c1.cast("double"), c2.cast("double")),
"floor_divide": pandas_udf( # type: ignore[call-overload]
lambda s1, s2: np.floor_divide(s1, s2), DoubleType()
),
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index 1f6ccd504cc4..53bdcc58e36c 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -154,6 +154,23 @@ class NumPyCompatTestsMixin:
np_func(psdf.value, psdf.bits), np_func(pdf.value, pdf.bits), almost=True
)
+ def test_np_float_power(self):
+ for pdf in (
+ pd.DataFrame({"base": [-64, -2, -1, 0, 1, 2, 64], "exponent": [-2, -1, 0, 1, 2, 3, 2]}),
+ pd.DataFrame(
+ {
+ "base": [-np.inf, -64.0, -2.0, -0.0, 0.0, 2.0, 64.0, np.inf, np.nan],
+ "exponent": [2.0, 3.0, -2.0, -3.0, -3.0, 0.5, -2.0, 2.0, 2.0],
+ }
+ ),
+ ):
+ psdf = ps.from_pandas(pdf)
+ self.assert_eq(
+ np.float_power(psdf.base, psdf.exponent),
+ np.float_power(pdf.base, pdf.exponent),
+ almost=True,
+ )
+
def test_np_spark_compat_series(self):
from pyspark.pandas.numpy_compat import unary_np_spark_mappings, binary_np_spark_mappings
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org