You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/02/12 22:27:10 UTC
[PR] [SPARK-55405][PYTHON][TESTS][FOLLOWUP] Add ARM overrides for float-to-int unsafe cast golden file mismatches [spark]
Yicong-Huang opened a new pull request, #54288:
URL: https://github.com/apache/spark/pull/54288
### What changes were proposed in this pull request?
Add hardcoded ARM (aarch64/arm64) overrides in `_overrides_unsafe()` for 34 float-to-integer unsafe cast golden file mismatches. Also rename `_version_overrides_safe/unsafe` to `_overrides_safe/unsafe` since overrides now cover both version and platform differences.
### Why are the changes needed?
The PyArrow array cast golden files are generated on x86. On ARM, unsafe float-to-integer casts produce different results due to IEEE 754 implementation-defined behavior:
- **ARM FCVT instructions** saturate on overflow: `inf` → type max, `-inf` → type min, `nan` → 0
- **x86 SSE/AVX** returns "integer indefinite" values (typically `0x80...0`)
- **Negative float → unsigned int**: ARM saturates to 0, x86 may wrap
This caused `test_scalar_cast_matrix_unsafe` to fail on Ubuntu 24.04 ARM CI runners with 34 mismatches across three categories:
- `float*:standard` → `uint*` (negative float -1.5 → unsigned int)
- `float*:special` → `int*` (inf/nan → signed int)
- `float*:special` → `uint*` (inf/nan → unsigned int)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Verified all 4 golden file tests pass on x86 (the ARM overrides are gated by `platform.machine()` check)
- The 34 override values are taken directly from the ARM CI failure output
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55405][PYTHON][TESTS][FOLLOW-UP] Add ARM overrides for float-to-int unsafe cast golden file mismatches [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #54288:
URL: https://github.com/apache/spark/pull/54288#issuecomment-3917908995
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55405][PYTHON][TESTS][FOLLOW-UP] Add ARM overrides for float-to-int unsafe cast golden file mismatches [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #54288: [SPARK-55405][PYTHON][TESTS][FOLLOW-UP] Add ARM overrides for float-to-int unsafe cast golden file mismatches
URL: https://github.com/apache/spark/pull/54288
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55405][PYTHON][TESTS][FOLLOW-UP] Add ARM overrides for float-to-int unsafe cast golden file mismatches [spark]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #54288:
URL: https://github.com/apache/spark/pull/54288#issuecomment-3900037511
cc @gaogaotiantian @zhengruifeng @ueshin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org