You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/06 11:26:33 UTC
(spark) branch branch-4.x updated: [SPARK-58580][PS][FOLLOWUP] Preserve ldexp zero and infinity inputs
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new aa1ad65503c5 [SPARK-58580][PS][FOLLOWUP] Preserve ldexp zero and infinity inputs
aa1ad65503c5 is described below
commit aa1ad65503c531e3b9567a8c58ea29c595076596
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Thu Aug 6 19:26:00 2026 +0800
[SPARK-58580][PS][FOLLOWUP] Preserve ldexp zero and infinity inputs
### What changes were proposed in this pull request?
This follow-up updates the native `np.ldexp` mapping to return a zero or infinite significand directly before evaluating `pow(2.0, exp)`. This avoids the invalid intermediate products `0.0 * Infinity` and `Infinity * 0.0`.
### Why are the changes needed?
For extreme exponents, the standalone power expression can overflow or underflow before multiplication. The prior native mapping therefore returned `NaN` for valid inputs such as `np.ldexp(0.0, 1024)` and `np.ldexp(np.inf, -1075)`, instead of preserving zero or infinity as NumPy does.
### Does this PR introduce _any_ user-facing change?
Yes. It corrects `np.ldexp` results for zero and infinite significands with exponents that overflow or underflow the power intermediate.
### How was this patch tested?
Added cases for positive and negative zero with exponent `1024`, plus positive and negative infinity with exponent `-1075`. The test also checks result sign bits.
- `python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_ldexp`
- `python/run-tests --testnames pyspark.pandas.tests.connect.test_parity_numpy_compat.NumPyCompatParityTests.test_np_ldexp`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex GPT-5
Closes #57820 from zhengruifeng/pandas-native-ldexp-followup-dev2.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
(cherry picked from commit 3afa7999adbb6ea9f50d689ccb01ba8cfb353e3e)
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/pandas/numpy_compat.py | 5 +++-
python/pyspark/pandas/tests/test_numpy_compat.py | 29 ++++++++++++++++++++++--
2 files changed, 31 insertions(+), 3 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py b/python/pyspark/pandas/numpy_compat.py
index 4ca221edd8ac..33538ef9de87 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -146,7 +146,10 @@ binary_np_spark_mappings = {
.otherwise(F.lit(1.0)),
"hypot": F.hypot,
"lcm": pandas_udf(lambda s1, s2: np.lcm(s1, s2), DoubleType()), # type: ignore[call-overload]
- "ldexp": lambda c1, c2: c1.cast("double") * F.pow(F.lit(2.0), c2),
+ "ldexp": lambda c1, c2: F.when(
+ c1.cast("double").isin(0.0, float("-inf"), float("inf")),
+ c1.cast("double"),
+ ).otherwise(c1.cast("double") * F.pow(F.lit(2.0), c2)),
# F.shiftleft accepts literal counts only; call_function also accepts a column.
# NumPy returns zero for counts outside an int64's bit width, unlike JVM shifts.
"left_shift": lambda c1, c2: F.when((c2 < 0) | (c2 >= 64), F.lit(0)).otherwise(
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index 0c19b8bfe47b..d95474f7fc07 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -188,13 +188,38 @@ class NumPyCompatTestsMixin:
1.0,
1.0,
1.0,
+ 0.0,
+ -0.0,
+ np.inf,
+ -np.inf,
+ ],
+ "exp": [
+ 2,
+ 3,
+ -2,
+ -3,
+ -3,
+ 0,
+ -2,
+ 2,
+ 2,
+ 2,
+ -1074,
+ -1075,
+ 1024,
+ 1024,
+ 1024,
+ -1075,
+ -1075,
],
- "exp": [2, 3, -2, -3, -3, 0, -2, 2, 2, 2, -1074, -1075, 1024],
}
)
psdf = ps.from_pandas(pdf)
- self.assert_eq(np.ldexp(psdf.x, psdf.exp), np.ldexp(pdf.x, pdf.exp), almost=True)
+ result = np.ldexp(psdf.x, psdf.exp)
+ expected = np.ldexp(pdf.x, pdf.exp)
+ self.assert_eq(result, expected, almost=True)
+ self.assert_eq(np.signbit(result.to_pandas()), np.signbit(expected))
def test_np_fmod(self):
for pdf in (
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org