You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/07/31 00:59:50 UTC
(spark) branch master updated: [SPARK-58445][PS] Fix NumPy isfinite and isinf mappings
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 24d7b0c31a25 [SPARK-58445][PS] Fix NumPy isfinite and isinf mappings
24d7b0c31a25 is described below
commit 24d7b0c31a2576bd2bb57aaf97c0c20dfbb98055
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Fri Jul 31 08:59:28 2026 +0800
[SPARK-58445][PS] Fix NumPy isfinite and isinf mappings
### What changes were proposed in this pull request?
Correct the pandas-on-Spark mappings for NumPy `isfinite` and `isinf`. Both predicates now check positive and negative infinity, and `isfinite` also excludes `NaN`. Add pandas-on-Spark parity coverage for finite values, both infinities, signed zero, and `NaN`.
### Why are the changes needed?
This is a longstanding pandas-on-Spark bug that has existed since pandas-on-Spark was first introduced. The existing mappings recognize only positive infinity. As a result, `np.isfinite` incorrectly returned true for negative infinity and `NaN`, while `np.isinf` incorrectly returned false for negative infinity.
### Does this PR introduce _any_ user-facing change?
Yes. `np.isfinite` and `np.isinf` on pandas-on-Spark objects now return NumPy-compatible results for negative infinity and `NaN`.
### How was this patch tested?
- Added pandas-on-Spark parity coverage for finite values, infinities, signed zero, and `NaN`.
- Ran `build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 -Phive package`.
- Ran `JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 SPARK_TESTING=1 SPARK_PREPEND_CLASSES=1 PYSPARK_PYTHON=.venv/bin/python PYSPARK_DRIVER_PYTHON=.venv/bin/python python/run-tests --testnames pyspark.pandas.tests.test_numpy_compat`.
- Ran `git diff --check`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
Closes #57586 from zhengruifeng/pandas-numpy-finite-ufuncs.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
python/pyspark/pandas/numpy_compat.py | 6 ++++--
python/pyspark/pandas/tests/test_numpy_compat.py | 2 ++
2 files changed, 6 insertions(+), 2 deletions(-)
diff --git a/python/pyspark/pandas/numpy_compat.py b/python/pyspark/pandas/numpy_compat.py
index 09194cbee5e0..febba91883bd 100644
--- a/python/pyspark/pandas/numpy_compat.py
+++ b/python/pyspark/pandas/numpy_compat.py
@@ -50,8 +50,10 @@ unary_np_spark_mappings = {
"frexp": lambda _: NotImplemented, # 'frexp' output lengths become different
# and it cannot be supported via pandas UDF.
"invert": F.bitwise_not,
- "isfinite": lambda c: c != float("inf"),
- "isinf": lambda c: c == float("inf"),
+ "isfinite": lambda c: F.coalesce(
+ ~(F.isnan(c) | (c == float("inf")) | (c == float("-inf"))), F.lit(False)
+ ),
+ "isinf": lambda c: F.coalesce((c == float("inf")) | (c == float("-inf")), F.lit(False)),
"isnan": F.isnan,
"isnat": lambda c: NotImplemented, # pandas-on-Spark and PySpark does not have Nat concept.
"log": F.log,
diff --git a/python/pyspark/pandas/tests/test_numpy_compat.py b/python/pyspark/pandas/tests/test_numpy_compat.py
index a3234ed4169d..432283cb5fe8 100644
--- a/python/pyspark/pandas/tests/test_numpy_compat.py
+++ b/python/pyspark/pandas/tests/test_numpy_compat.py
@@ -99,6 +99,8 @@ class NumPyCompatTestsMixin:
(np.fabs, [np.iinfo(np.int64).min, -2, 0, 2]),
(np.fabs, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.invert, [np.iinfo(np.int64).min, -2, -1, 0, 1, 2, np.iinfo(np.int64).max]),
+ (np.isfinite, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
+ (np.isinf, [-np.inf, -64.0, -0.0, 0.0, 64.0, np.inf, np.nan]),
(np.negative, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.positive, [-np.inf, -64.0, -2.0, 0.0, 2.0, 64.0, np.inf, np.nan]),
(np.rad2deg, [-np.inf, -64.0, -np.pi, 0.0, np.pi, 64.0, np.inf, np.nan]),
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org