You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/08/10 04:12:34 UTC
(spark) branch master updated: [SPARK-58553][PS][FOLLOWUP] Preserve signed zero in NumPy fmax and fmin
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 32a65fbe5491 [SPARK-58553][PS][FOLLOWUP] Preserve signed zero in NumPy fmax and fmin
32a65fbe5491 is described below
commit 32a65fbe54913282c2f3579e9e443dd3d63f7fd4
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Aug 10 12:12:20 2026 +0800
[SPARK-58553][PS][FOLLOWUP] Preserve signed zero in NumPy fmax and fmin
### What changes were proposed in this pull request?
This follow-up fixes the native `np.fmax` and `np.fmin` mappings to preserve the first operand when the operands compare equal. This preserves NumPy signed-zero behavior while retaining the existing NaN handling and native Spark expressions.
### Why are the changes needed?
Spark `greatest` and `least` order `-0.0` and `+0.0`, whereas NumPy retains the first operand for an equal-value tie. Consequently, the prior mapping returned the wrong signed zero for `fmax(-0.0, 0.0)` and `fmin(0.0, -0.0)`.
### Does this PR introduce _any_ user-facing change?
Yes. It corrects the sign of zero for the affected `np.fmax` and `np.fmin` inputs.
### How was this patch tested?
Added signbit-based regression coverage for both signed-zero operand orders.
- `python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin`
- `python/run-tests --testnames pyspark.pandas.tests.connect.test_parity_numpy_compat.NumPyCompatParityTests.test_np_fmax_fmin`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex GPT-5
Closes #57819 from zhengruifeng/pandas-native-fmax-fmin-followup-dev2.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
python/pyspark/pandas/numpy_compat.py | 3 ++-
python/pyspark/pandas/tests/test_numpy_compat.py | 7 +++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/python/pyspark/pandas/numpy_compat.py b/python/pyspark/pandas/numpy_compat.py
index 33538ef9de87..3df41058f91b 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -132,9 +132,10 @@ binary_np_spark_mappings = {
),
"fmax": lambda c1, c2: F.when(F.isnan(c1.cast("double")), c2)
.when(F.isnan(c2.cast("double")), c1)
+ .when(c1 == c2, c1)
.otherwise(F.greatest(c1, c2))
.cast("double"),
- "fmin": lambda c1, c2: F.least(c1, c2).cast("double"),
+ "fmin": lambda c1, c2: F.when(c1 == c2, c1).otherwise(F.least(c1, c2)).cast("double"),
"fmod": _fmod_func,
"gcd": pandas_udf(lambda s1, s2: np.gcd(s1, s2), DoubleType()), # type: ignore[call-overload]
"heaviside": lambda c1, c2: F.when(
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index d95474f7fc07..ab4537498016 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -255,12 +255,19 @@ class NumPyCompatTestsMixin:
"x2": [2.0, np.nan, np.nan, np.inf, -np.inf, 0.0, -0.0, np.inf, -np.inf],
}
),
+ pd.DataFrame({"x1": [-0.0, 0.0], "x2": [0.0, -0.0]}),
):
psdf = ps.from_pandas(pdf)
for np_func in (np.fmax, np.fmin):
result = np_func(psdf.x1, psdf.x2)
expected = np_func(pdf.x1, pdf.x2)
self.assert_eq(result, expected, almost=True)
+ # NumPy's vectorized implementation may select either zero operand, whereas
+ # its scalar implementation consistently selects the first one.
+ expected_signbit = pd.Series(
+ [np.signbit(np_func(x1, x2)) for x1, x2 in zip(pdf.x1, pdf.x2)]
+ )
+ self.assert_eq(np.signbit(result.to_pandas()), expected_signbit)
def test_np_heaviside(self):
for pdf in (
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org