You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/06/25 22:35:59 UTC
[PR] [DO-NOT-MERGE][PS][PYTHON] Make as_spark_type handle numpy>=2.5 NDArray annotations [spark]
HyukjinKwon opened a new pull request, #56788:
URL: https://github.com/apache/spark/pull/56788
### What changes were proposed in this pull request?
`as_spark_type` detected `numpy.typing.NDArray[...]` annotations via `tpe.__origin__ is np.ndarray` and, when that did not match, fell through to `issubclass(tpe.__origin__, list)`.
numpy 2.5 changed the internal structure of `NDArray` aliases so that `tpe.__origin__` is no longer the bare `numpy.ndarray` class but a subscripted generic alias (which is not a class). As a result the NDArray branch is skipped and `issubclass(tpe.__origin__, list)` raises `TypeError: issubclass() arg 1 must be a class` (surfaced to the user as `TypeError: Cannot interpret 'NDArray[int]' as a data type`).
This PR:
- adds a `_numpy_ndarray_element_type` helper that unwraps the `__origin__` chain to detect `numpy.ndarray` robustly across numpy versions, then scans the type arguments for the `np.dtype[...]` scalar and returns it;
- guards the list branch with `isclass(...)` so the `issubclass` call cannot raise on a non-class origin.
### Why are the changes needed?
The scheduled `Build / Python-only (Python 3.12, MacOS26)` job installs numpy 2.5.0 and fails `pyspark.pandas` and `pyspark.pandas.connect` `test_apply_batch_with_type` with the `TypeError` above (e.g. apache/spark Actions run 28064134730). The breakage blocks that CI job and breaks numpy `NDArray` return-type inference for pandas-on-Spark `apply_batch`/`transform_batch` under numpy >= 2.5.
### Does this PR introduce _any_ user-facing change?
No. It restores correct Spark type inference for numpy `NDArray` return annotations under numpy >= 2.5; behavior on older numpy is unchanged.
### How was this patch tested?
Existing parametrized coverage in `pyspark.pandas.tests.test_typedef` exercises `as_spark_type(NDArray[...])` and `pandas_on_spark_type(NDArray[...])` over numpy/Python scalar types; that is the regression test for this path.
The detection/extraction helper was additionally verified against numpy 2.3.5, numpy 2.4.6, and a simulated numpy-2.5 alias structure (where `__origin__` is a non-class generic alias): `NDArray[int]`/`NDArray[np.float64]` resolve to the correct scalar; `list[int]`, the bare `np.ndarray`, and plain `int` resolve to `None` (falling through to the existing branches). Validated on macOS-26 with numpy 2.5.0 via GitHub Actions on the fork (the `Build / Python-only (Python 3.12, MacOS26)` workflow).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][PS][PYTHON] Make as_spark_type handle numpy>=2.5 NDArray annotations [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #56788: [DO-NOT-MERGE][PS][PYTHON] Make as_spark_type handle numpy>=2.5 NDArray annotations
URL: https://github.com/apache/spark/pull/56788
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][PS][PYTHON] Make as_spark_type handle numpy>=2.5 NDArray annotations [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56788:
URL: https://github.com/apache/spark/pull/56788#issuecomment-4804787736
Closing in favor of #56789 (agent6), which is the correct fix. I verified locally (Python 3.12) that numpy 2.5 declares `NDArray` as a PEP 695 `type` alias, so `NDArray[int].__origin__` is a `TypeAliasType` (with `__value__`) and `__args__` is `(int,)`. The approach in this PR (unwrapping `__origin__` / scanning for `np.dtype[...]`) dead-ends on the `TypeAliasType` and returns `None`, so it would not actually resolve the element type. #56789 detects the `TypeAliasType` and resolves it correctly. Superseded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [DO-NOT-MERGE][PS][PYTHON] Make as_spark_type handle numpy>=2.5 NDArray annotations [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #56788:
URL: https://github.com/apache/spark/pull/56788#issuecomment-4804757816
cc @gaogaotiantian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org