(spark) branch branch-4.x updated: [SPARK-58553][PS][TEST][FOLLOWUP] Skip NumPy comparison tests where NumPy's own result is platform-dependent

[ru...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new bdaaf430638c [SPARK-58553][PS][TEST][FOLLOWUP] Skip NumPy comparison tests where NumPy's own result is platform-dependent
bdaaf430638c is described below

commit bdaaf430638c2ca6d10815e1a7ba9322bf4f1959
Author: Spenser Sun <hs...@gmail.com>
AuthorDate: Tue Aug 25 08:48:40 2026 +0800

    [SPARK-58553][PS][TEST][FOLLOWUP] Skip NumPy comparison tests where NumPy's own result is platform-dependent
    
    ### What changes were proposed in this pull request?
    
    Skip the NumPy-comparison tests for `reciprocal` and `fmax`/`fmin` unless running on Linux x86-64 with NumPy >= 2.3.0, and exclude `reciprocal` from the random-integer sweeps (`test_np_spark_compat_series` / `_frame`) elsewhere. Test-only change; `numpy_compat.py` is untouched.
    
    This is an alternative to #58019, which instead changed the implementation to fall back to `pandas_udf` off that platform.
    
    ### Why are the changes needed?
    
    `np.reciprocal(int 0)` and the `fmax`/`fmin` signed-zero tie are unspecified by C/IEEE, so NumPy's own answer varies by CPU architecture (x86-64 casts the intermediate `+inf` to `INT64_MIN`, aarch64 saturates to `INT64_MAX`) and by NumPy version (the scalar tie returns the first operand from 2.3.0 on, the second before).
    
    pandas-on-Spark returns one fixed value everywhere, so NumPy is only a valid reference on the platform it was verified against. The scheduled ARM and macOS jobs fail on `test_np_fmax_fmin`, `test_np_reciprocal_integer`, and intermittently `test_np_spark_compat_series`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Tests only.
    
    ### How was this patch tested?
    
    Existing tests. On Linux x86-64 with NumPy 2.4.1 both tests still run and pass; with `platform.machine`, `platform.system`, and `np.__version__` patched to macOS arm64, Linux aarch64, and NumPy 1.23.2, they are skipped as intended.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #58257 from Spenserrrr/numpy-crossplatform-skip-tests.
    
    Authored-by: Spenser Sun <hs...@gmail.com>
    Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
    (cherry picked from commit 204a73d8f31e0c35aec87e73af77bcbe5df27152)
    Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
 python/pyspark/pandas/tests/test_numpy_compat.py | 26 ++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index 17201690cee2..4e9c082cb2c7 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -15,15 +15,36 @@
 # limitations under the License.
 #
 
+import platform
+import unittest
+
 import numpy as np
 import pandas as pd
 
 from pyspark import pandas as ps
+from pyspark.loose_version import LooseVersion
 from pyspark.pandas import set_option, reset_option
 from pyspark.sql import functions as F
 from pyspark.testing.pandasutils import PandasOnSparkTestCase
 
 
+# np.reciprocal(int 0) and the fmax/fmin signed-zero tie are unspecified by C/IEEE, so NumPy's
+# own answer varies by CPU architecture and NumPy version. pandas-on-Spark returns one fixed
+# value everywhere, which matches NumPy only on the environment it was verified against, so the
+# tests comparing the two run only there.
+_numpy_matches_spark = (
+    platform.system() == "Linux"
+    and platform.machine() == "x86_64"
+    and LooseVersion(np.__version__) >= LooseVersion("2.3.0")
+)
+_skip_if_numpy_differs = unittest.skipIf(
+    not _numpy_matches_spark,
+    "NumPy's reciprocal(int 0) and fmax/fmin signed-zero tie vary by architecture and NumPy "
+    "version, while pandas-on-Spark returns one fixed value that matches NumPy only on "
+    "Linux x86_64 with NumPy >= 2.3.0",
+)
+
+
 class NumPyCompatTestsMixin:
     @classmethod
     def setUpClass(cls):
@@ -43,6 +64,9 @@ class NumPyCompatTestsMixin:
         "log10",  # flaky
         "log1p",  # flaky
     ]
+    # The sweeps below draw random integers including 0, where reciprocal diverges.
+    if not _numpy_matches_spark:
+        blacklist = blacklist + ["reciprocal"]
 
     @property
     def pdf(self):
@@ -139,6 +163,7 @@ class NumPyCompatTestsMixin:
 
                 self.assert_eq(np_func(psdf.a), np_func(pdf.a), almost=True)
 
+    @_skip_if_numpy_differs
     def test_np_reciprocal_integer(self):
         # np.reciprocal on an integer column does integer division (truncated
         # toward zero): 1 -> 1, -1 -> -1, and every other magnitude -> 0. The
@@ -428,6 +453,7 @@ class NumPyCompatTestsMixin:
                 self.assert_eq(result, expected, almost=True)
                 self.assert_eq(np.signbit(result.to_pandas()), np.signbit(expected))
 
+    @_skip_if_numpy_differs
     def test_np_fmax_fmin(self):
         for pdf in (
             pd.DataFrame({"x1": [-2, -1, 0, 1, 2], "x2": [2, 1, 0, -1, -2]}),


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org