You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/05 06:51:09 UTC
(spark) branch master updated: [SPARK-58580][PS] Use native Spark function for NumPy ldexp
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fdd586ef0c90 [SPARK-58580][PS] Use native Spark function for NumPy ldexp
fdd586ef0c90 is described below
commit fdd586ef0c903e5b5616813a2f1bb24a03ef7886
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Wed Aug 5 14:50:57 2026 +0800
[SPARK-58580][PS] Use native Spark function for NumPy ldexp
### What changes were proposed in this pull request?
Replaces the pandas UDF used for `np.ldexp` in the pandas API on Spark with the native Spark SQL expression `x * pow(2.0, exp)`.
Adds coverage for finite values, infinities, NaN, signed zero, and exponent overflow and underflow.
### Why are the changes needed?
Using native Spark expressions avoids pandas UDF serialization and lets Spark optimize and code-generate this operation while preserving NumPy-compatible behavior.
### Does this PR introduce _any_ user-facing change?
No. This is a behavior-preserving implementation change.
### How was this patch tested?
`source ~/.zshrc && conda run -n spark-dev-313 python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_ldexp`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57780 from zhengruifeng/pandas-native-ldexp-dev2.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
python/pyspark/pandas/numpy_compat.py | 4 +---
python/pyspark/pandas/tests/test_numpy_compat.py | 25 ++++++++++++++++++++++++
2 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py b/python/pyspark/pandas/numpy_compat.py
index c109df5e9b82..4894fbfbec7b 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -127,9 +127,7 @@ binary_np_spark_mappings = {
.otherwise(F.lit(1.0)),
"hypot": F.hypot,
"lcm": pandas_udf(lambda s1, s2: np.lcm(s1, s2), DoubleType()), # type: ignore[call-overload]
- "ldexp": pandas_udf( # type: ignore[call-overload]
- lambda s1, s2: np.ldexp(s1, s2), DoubleType()
- ),
+ "ldexp": lambda c1, c2: c1.cast("double") * F.pow(F.lit(2.0), c2),
# F.shiftleft accepts literal counts only; call_function also accepts a column.
# NumPy returns zero for counts outside an int64's bit width, unlike JVM shifts.
"left_shift": lambda c1, c2: F.when((c2 < 0) | (c2 >= 64), F.lit(0)).otherwise(
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index 5797337945e6..b4324ea208b3 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -171,6 +171,31 @@ class NumPyCompatTestsMixin:
almost=True,
)
+ def test_np_ldexp(self):
+ pdf = pd.DataFrame(
+ {
+ "x": [
+ -np.inf,
+ -64.0,
+ -2.0,
+ -0.0,
+ 0.0,
+ 1.0,
+ 2.0,
+ 64.0,
+ np.inf,
+ np.nan,
+ 1.0,
+ 1.0,
+ 1.0,
+ ],
+ "exp": [2, 3, -2, -3, -3, 0, -2, 2, 2, 2, -1074, -1075, 1024],
+ }
+ )
+ psdf = ps.from_pandas(pdf)
+
+ self.assert_eq(np.ldexp(psdf.x, psdf.exp), np.ldexp(pdf.x, pdf.exp), almost=True)
+
def test_np_fmax_fmin(self):
for pdf in (
pd.DataFrame({"x1": [-2, -1, 0, 1, 2], "x2": [2, 1, 0, -1, -2]}),
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org