You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Spenserrrr (via GitHub)" <gi...@apache.org> on 2026/08/15 09:42:59 UTC

[PR] [SPARK-58553][PS][FOLLOWUP] Gate native fmax/fmin/reciprocal to verified platform, else pandas_udf [spark]

Spenserrrr opened a new pull request, #58019:
URL: https://github.com/apache/spark/pull/58019

   ### What changes were proposed in this pull request?
   
   This is a follow-up to the native conversions of `fmax`/`fmin` (SPARK-58553) and `reciprocal` (SPARK-58465, SPARK-58646) in `pyspark.pandas.numpy_compat`. Those conversions replaced the `pandas_udf` fallbacks with native Spark expressions, but for these three functions the native expression reproduces behavior that C/IEEE leave unspecified, so it only matches NumPy on the platform and NumPy version where that behavior was pinned down.
   
   This PR restricts the native path to the verified platform and otherwise falls back to the real NumPy ufunc via `pandas_udf` (which matches NumPy on any platform/version by construction). Two module-level predicates express the two independent axes of divergence:
   
   - `_native_platform = platform.system() == "Linux" and platform.machine() == "x86_64"`
   - `_numpy_tie_returns_first = LooseVersion(np.__version__) >= LooseVersion("2.3.0")`
   
   `reciprocal` gates on `_native_platform` only (its divergence is architecture-specific, not version-specific); `fmax`/`fmin` gate on `_native_platform and _numpy_tie_returns_first`.
   
   ### Why are the changes needed?
   
   The scheduled ARM (`ubuntu-24.04-arm`, aarch64) and macOS jobs fail on `test_np_fmax_fmin` and `test_np_reciprocal_integer`, because NumPy's own result for these operations is not portable:
   
   - `reciprocal` of an integer `0` is computed through a float→int cast of `+inf`, which is undefined behavior in C and resolved per CPU architecture: x86-64 yields `INT64_MIN`, aarch64 yields `INT64_MAX`.
   - The `fmax`/`fmin` signed-zero tie (equal operands differing only in the sign of zero, e.g. `+0.0` vs `-0.0`) is unspecified by C99. NumPy's scalar result returns the first operand from 2.3.0 on and the second before, and it also differs across architectures.
   
   The native expressions were tuned to the x86-64 Linux / NumPy >= 2.3.0 result, so they diverge on other architectures and older NumPy. The minimum-dependencies job (older NumPy) was addressed for `fmax`/`fmin` in an earlier SPARK-58553 follow-up, but the underlying cross-platform (architecture) divergence remained on aarch64 and macOS, and `reciprocal` was affected as well.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. On the verified platform the native expression is unchanged. On other platforms/versions the result now matches the installed NumPy (via the UDF) instead of a platform-specific value, restoring behavior-preserving parity with NumPy. There is no API change.
   
   ### How was this patch tested?
   
   No test changes were needed: the existing `test_np_fmax_fmin`, `test_np_reciprocal_integer`, and the `NumPyCompatTests` compat sweep use the installed NumPy as the reference, so the mapping now tracks NumPy on every platform.
   
   Verified on GitHub Actions across all affected environments — ARM (`ubuntu-24.04-arm`, aarch64), macOS (`macos-26`), the minimum-dependencies build (older NumPy), and the regular x86-64 Linux build — all green. Also ran the full `NumPyCompatTests` locally on x86-64 Linux with both the native path and the UDF fallback path exercised, both passing.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Anthropic)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58553][PS][FOLLOWUP] Gate native fmax/fmin/reciprocal to verified platform, else pandas_udf [spark]

Posted by "Spenserrrr (via GitHub)" <gi...@apache.org>.
Spenserrrr closed pull request #58019: [SPARK-58553][PS][FOLLOWUP] Gate native fmax/fmin/reciprocal to verified platform, else pandas_udf
URL: https://github.com/apache/spark/pull/58019


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58553][PS][FOLLOWUP] Gate native fmax/fmin/reciprocal to verified platform, else pandas_udf [spark]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58019:
URL: https://github.com/apache/spark/pull/58019#issuecomment-5320732805

   I think there are quite many compatibility dimensions we would need to maintain, OS, instruction set, NumPy version, etc. I’m wondering whether it’s worth the effort to replace Pandas UDFs with native functions if we can support only a subset of configurations? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58553][PS][FOLLOWUP] Gate native fmax/fmin/reciprocal to verified platform, else pandas_udf [spark]

Posted by "Spenserrrr (via GitHub)" <gi...@apache.org>.
Spenserrrr commented on PR #58019:
URL: https://github.com/apache/spark/pull/58019#issuecomment-5301786616

   Hi @zhengruifeng @Yicong-Huang! This is a PR fixing fmax/fmin and reciprocal failures on macOS and ARM CI. Could you take a look when you have time? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-58553][PS][FOLLOWUP] Gate native fmax/fmin/reciprocal to verified platform, else pandas_udf [spark]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #58019:
URL: https://github.com/apache/spark/pull/58019#issuecomment-5306749245

   Thank you @Spenserrrr! Added @zhengruifeng here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org