You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/08/21 20:36:39 UTC
[PR] [SPARK-58646][PS][FOLLOWUP] Fix NumPy reciprocal parity for decimal, boolean, and narrower integer columns [spark]
Yicong-Huang opened a new pull request, #58218:
URL: https://github.com/apache/spark/pull/58218
### What changes were proposed in this pull request?
This is a follow-up of #57856 (SPARK-58646), which replaced the scalar pandas UDF fallback of `np.reciprocal` with native Spark expressions for non-floating-point inputs.
That change routed every non-`float`/`double` dtype through a single integer branch that hard-codes the int64 minimum as the divide-by-zero sentinel and casts the quotient through `long`. This does not match the previous pandas UDF (`np.reciprocal` applied to the pandas `Series`) for decimals, booleans, and narrower integers:
| dtype | input | previous UDF (`np.reciprocal` -> Double) | merged native (#57856) |
|---|---|---|---|
| int64 (bigint) | `0` | `-9.2e18` (int64 min) | `-9.2e18` (unchanged) |
| int32 (int) | `0` | `-2147483648` (int32 min) | `-9.2e18` |
| int8/int16 (tinyint/smallint) | `0` | `0` (numpy `1 // 0` does not overflow on narrow widths) | `-9.2e18` |
| boolean | `False` | `0.0` (numpy promotes bool to int8: `True -> 1`, `False -> 0`) | `-9.2e18` |
| decimal | `2.5` | `0.4` (numpy takes a true floating reciprocal) | `0.0` (truncated to long) |
This PR restores parity:
- Decimal inputs now flow through the floating-point reciprocal branch (`typeof` starts with `decimal`), since numpy computes a true reciprocal for them. A decimal `0` (which the old UDF could not handle -- `np.reciprocal(Decimal('0'))` raises `DivisionByZero`) now maps to `inf`, consistent with the floating-point branch.
- The integer/boolean branch now picks the divide-by-zero sentinel by column width -- int32 minimum for `int`, int64 minimum for `bigint`, and `0` for the narrower widths (`tinyint`, `smallint`, and `boolean` promoted to int8) -- and casts through `long` so boolean and narrower integers can take part in the division.
int64 columns, the only case exercised by the original PR, are unchanged.
### Why are the changes needed?
The merged native expression regressed the observable pandas-on-Spark behavior for decimal, boolean, and narrower-integer columns relative to the pandas UDF it replaced. This restores parity so that `np.reciprocal` produces the same results as before across all supported dtypes.
### Does this PR introduce _any_ user-facing change?
No. #57856 is unreleased (master only), so this only fixes an unreleased regression before it ships; there is no change relative to any released Spark version.
### How was this patch tested?
Added `test_np_reciprocal_non_default_dtypes` in `python/pyspark/pandas/tests/test_numpy_compat.py`, inherited by the Spark Connect parity suite, asserting `np.reciprocal(psser)` equals `np.reciprocal(pdf)` for int8/int16/int32, boolean, and decimal columns (covering positive, negative, and the per-width zero-overflow sentinel).
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58646][PS][FOLLOWUP] Fix NumPy reciprocal parity for decimal, boolean, and narrower integer columns [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #58218:
URL: https://github.com/apache/spark/pull/58218#issuecomment-5423658773
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/34aad86c44b68b49ab937c08bcf395dd0082a838
- merged into branch-4.x https://github.com/apache/spark/commit/3869fb626680df0675d0408ed2ffa503ded85fbe
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58646][PS][FOLLOWUP] Fix NumPy reciprocal parity for decimal, boolean, and narrower integer columns [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed pull request #58218: [SPARK-58646][PS][FOLLOWUP] Fix NumPy reciprocal parity for decimal, boolean, and narrower integer columns
URL: https://github.com/apache/spark/pull/58218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org