You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/07/24 01:32:20 UTC
(spark) branch branch-4.x updated: [SPARK-58249][SPARK-58234][PS][TEST][FOLLOWUP] Test native NumPy ufuncs with special values
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new cd0334f9e2cc [SPARK-58249][SPARK-58234][PS][TEST][FOLLOWUP] Test native NumPy ufuncs with special values
cd0334f9e2cc is described below
commit cd0334f9e2ccfe897f348e90ab31ba01396241d4
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Fri Jul 24 09:31:40 2026 +0800
[SPARK-58249][SPARK-58234][PS][TEST][FOLLOWUP] Test native NumPy ufuncs with special values
### What changes were proposed in this pull request?
Add pandas-on-Spark parity coverage for the native NumPy ufunc mappings added in #57397 and #57417. The tests compare pandas-on-Spark results with pandas/NumPy for invalid mathematical inputs, `NaN`, and positive and negative infinity. They retain the `int64` minimum-value case for `np.fabs`.
### Why are the changes needed?
The mappings now execute native Spark SQL functions instead of scalar pandas UDFs. IEEE special values and invalid function domains are where the two implementations can differ, so the coverage verifies that the native results remain NumPy-compatible.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Added pandas-on-Spark comparisons with pandas/NumPy for invalid inputs, `NaN`, and infinities.
- Ran `build/sbt -Phive package`.
- Ran `SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1 PYSPARK_PYTHON=$CONDA_PREFIX/bin/python PYSPARK_DRIVER_PYTHON=$CONDA_PREFIX/bin/python bin/pyspark pyspark.pandas.tests.test_numpy_compat` in the `spark-dev-313` conda environment.
- Ran `SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1 PYSPARK_PYTHON=$CONDA_PREFIX/bin/python PYSPARK_DRIVER_PYTHON=$CONDA_PREFIX/bin/python python/run-tests --testnames "pyspark.pandas.tests.test_numpy_compat NumPyCompatTests.test_np_math_functions"` in the `spark-dev-313` conda environment.
- Ran `git diff --check`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57458 from zhengruifeng/pandas-native-math-special-values-dev2.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
(cherry picked from commit 94d47152ec053fb2ee7bc9fdf257e6c95502940d)
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/pandas/tests/test_numpy_compat.py | 35 +++++++++++++-----------
1 file changed, 19 insertions(+), 16 deletions(-)
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index 2864f8637796..ed5e53a8bd68 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -87,27 +87,30 @@ class NumPyCompatTestsMixin:
def test_np_math_functions(self):
for np_func, values in (
- (np.arccosh, [1.0, 2.0, 4.0]),
- (np.arcsinh, [-2.0, 0.0, 2.0]),
- (np.arctanh, [-0.5, 0.0, 0.5]),
- (np.cosh, [-2.0, 0.0, 2.0]),
- (np.deg2rad, [-180.0, 0.0, 180.0]),
- (np.exp2, [-2.0, 0.0, 2.0]),
+ (np.arccosh, [-np.inf, -1.0, 0.0, 1.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.arcsinh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (
+ np.arctanh,
+ [-np.inf, -64.0, -1.0, 0.0, 1.0, 64.0, np.inf, np.nan],
+ ),
+ (np.cosh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.deg2rad, [-np.inf, -64.0, -180.0, 0.0, 180.0, 64.0, np.inf, np.nan]),
+ (np.exp2, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.fabs, [np.iinfo(np.int64).min, -2, 0, 2]),
- (np.negative, [-2.0, 0.0, 2.0]),
- (np.positive, [-2.0, 0.0, 2.0]),
- (np.rad2deg, [-np.pi, 0.0, np.pi]),
- (np.sign, [-2.0, -0.0, 0.0, 2.0, np.nan]),
- (np.sinh, [-2.0, 0.0, 2.0]),
- (np.square, [-2.0, 0.0, 2.0]),
- (np.tanh, [-2.0, 0.0, 2.0]),
+ (np.fabs, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf, np.nan]),
+ (np.sign, [-np.inf, -64.0, -2.0, -0.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.sinh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.square, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
+ (np.tanh, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
):
- with self.subTest(name=np_func.__name__):
+ with self.subTest(name=np_func.__name__, values=values):
pdf = pd.DataFrame({"a": values})
psdf = ps.from_pandas(pdf)
- result = np_func(psdf.a)
- self.assert_eq(result, np_func(pdf.a), almost=True)
+ self.assert_eq(np_func(psdf.a), np_func(pdf.a), almost=True)
def test_np_spark_compat_series(self):
from pyspark.pandas.numpy_compat import unary_np_spark_mappings, binary_np_spark_mappings
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org