You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "jiayuasu (via GitHub)" <gi...@apache.org> on 2026/10/12 09:06:08 UTC
[PR] [CI] Run the Python tests on NumPy 2 for Spark 4 [sedona]
jiayuasu opened a new pull request, #3445:
URL: https://github.com/apache/sedona/pull/3445
## Did you read the Contributor Guide?
- Yes, I have read the [Contributor Rules](https://sedona.apache.org/latest/community/rule/) and [Contributor Development Guide](https://sedona.apache.org/latest/community/develop/)
## Is this PR related to a ticket?
- No:
- this is a CI update. The PR name follows the format `[CI] my subject`
## What changes were proposed in this PR?
The Python build on master started failing on the Spark 4.1.1 / Python 3.11 job (https://github.com/apache/sedona/actions/runs/38216164257/job/114701289570). pyarrow 26.0.0 was released on 2026-10-09. It needs NumPy 2 at import time but does not declare numpy as a dependency, and our dev group pins `numpy<2`, so uv installs pyarrow 26.0.0 next to numpy 1.26.4. Every test module that imports pyarrow then fails at collection with `ImportError: pyarrow requires NumPy 2.0 or newer, found 1.26.4`.
Instead of capping pyarrow, this PR lets the Spark 4 jobs run on NumPy 2:
- Drop `numpy<2` and `scipy<=1.10.0` from the `dev` group in `python/pyproject.toml`. Both came over unchanged from the old Pipfile. The scipy cap worked around an old libpysal release, and scipy 1.10 has no NumPy 2 builds, so it held NumPy back as well.
- Keep `numpy<2` for the Spark 3.5 jobs in `python.yml`. pyspark 3.5.0 through 3.5.8 still use names removed in NumPy 2 (`np.float_`, `np.string_`, `np.unicode_`, `np.NaN` in `pyspark.pandas`), and `sedona.spark.geopandas` depends on `pyspark.pandas`. This works the same way as the existing `shapely<2` pin for the Shapely 1 job.
The Sedona package itself needs no change. Its runtime dependencies never pinned numpy, the C serde extension does not use the NumPy C API, and `ruff --select NPY201` finds no NumPy 2 removed APIs under `python/`.
Resolved versions per job after this change:
| Job | numpy | scipy | Other changes |
|---|---|---|---|
| Spark 4.1.1, Python 3.11 | 1.26.4 → 2.4.6 | 1.10.0 → 1.17.1 | pyarrow 26.0.0, geopandas 1.2.0 |
| Spark 4.0.0, Python 3.10 | 1.26.4 → 2.2.6 | 1.10.0 → 1.15.3 | |
| Spark 3.5.0, Python 3.9 | 1.26.4 | 1.10.0 → 1.13.1 | |
| Spark 3.5.0, Python 3.8, Shapely 1 | 1.24.4 | 1.10.0 → 1.10.1 | |
The PyFlink workflow does not change. `apache-flink==1.20.1` requires `apache-beam<2.49`, which keeps numpy below 1.25 on its own.
## How was this patch tested?
I reproduced each job's resolution locally by applying the same `uv add` / `uv remove` steps from `python.yml` to `pyproject.toml`, then ran the tests in venvs built from those pins:
- Full `pytest tests` on Spark 4.1.1 / Python 3.11 with numpy 2.4.6, pyarrow 26.0.0, geopandas 1.2.0 and scipy 1.17.1: 2569 passed, 17 skipped. The last green run on master with numpy 1.26.4 had the same counts.
- `tests/stats` (the only tests that use scipy, esda and libpysal) on Spark 3.5.0: 30 passed on Python 3.9 with scipy 1.13.1 and libpysal 4.8.1, and 30 passed on Python 3.8 with scipy 1.10.1 and libpysal 4.7.0.
- `tests/utils/test_geomserde_speedup.py` with Shapely 1.8.5 and numpy 2.1.3, which is what the Shapely 1.8 step of `python-extension.yml` now resolves: 36 passed, 26 skipped, the same as with numpy 1.26.4.
## Did this PR include necessary documentation updates?
- No, this PR does not affect any public API so no need to change the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CI] Run the Python tests on NumPy 2 for Spark 4 [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu merged PR #3445:
URL: https://github.com/apache/sedona/pull/3445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org