You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2025/10/16 02:21:58 UTC
[PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
zhengruifeng opened a new pull request, #52633:
URL: https://github.com/apache/spark/pull/52633
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
Fix scheduled job for numpy 2.1.3
### Why are the changes needed?
to fix https://github.com/apache/spark/actions/runs/18538043179/job/52838303733
it was caused by a bug in 19.0.0, see https://github.com/apache/arrow/issues/45283
### Does this PR introduce _any_ user-facing change?
no, infra-only
### How was this patch tested?
PR builder with
```
default: '{"PYSPARK_IMAGE_TO_TEST": "python-311", "PYTHON_TO_TEST": "python3.11"}'
```
see https://github.com/zhengruifeng/spark/actions/runs/18527303212/job/52801019275
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #52633:
URL: https://github.com/apache/spark/pull/52633#issuecomment-3409229955
merged to master to restore the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #52633:
URL: https://github.com/apache/spark/pull/52633#issuecomment-3409032603
> @zhengruifeng, I have a silly question about Python deps management - I see that many Python deps are declared without a version, or with a range version(half-bounded, e.g. `foo>=1.0` or `bar<2.0`). Silently upgrading 3rd libs may introduce breaking changes (especially for major version bumping)/bugs.
>
> This means that if we do not specify the dependency version, or only specify the lower bound of the dependency version, PySpark may not work once a new major version of the dependency is released. This becomes a problem if users want to use older Spark versions (in practice, EOLed versions of Spark are used widely and upgrading is not timely).
>
> I wonder if Spark can pin all Python deps in a fixed version(or at least a bounded range version, e.g. `foo>=1.0,<=2.3`), this clearly shows the versions of Spark that have been fully tested.
@pan3793 the reason to use lower bounds `foo>=1.0` in most places is to eagerly testing spark against latest packages (need to trigger the refresh of cached images), when spark get broken with new versions we set the upper bound (e.g. https://github.com/apache/spark/commit/22d2eb33f01ddff30e4f37fc529912372b35019b) and restore it once the issue get resolved (e.g. https://github.com/apache/spark/commit/47574ba833af370bc8527b352f846d4e3a645d77)
currently, most workflows are testing against latests version; and we have two workflow against the minimum versions in which the versions of key packages (numpy/pyarrow/pandas) are pinned
- https://github.com/apache/spark/actions/workflows/build_python_minimum.yml
- https://github.com/apache/spark/actions/workflows/build_python_ps_minimum.yml
But I personally think maybe we should use a fixed version `foo=1.2.3` or `foo>=1.0,<=2.3` in the officially released images
@dongjoon-hyun @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52633:
URL: https://github.com/apache/spark/pull/52633#issuecomment-3409261675
Thank you. In that case, it looks okay to me, too. It doesn't need us to pay more attention.
> I am not sure since it is a pyarrow bug introduced in 19.0.0 and fixed in 19.0.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #52633:
URL: https://github.com/apache/spark/pull/52633#issuecomment-3409036437
> For this one, do you think we need to document somewhere this incompatibility because our minimum `numpy` is still `1.22` and `pyarrow` is `15.0.0`.
>
> > it was caused by a bug in 19.0.0, see [apache/arrow#45283](https://github.com/apache/arrow/issues/45283)
@dongjoon-hyun I am not sure since it is a pyarrow bug introduced in 19.0.0 and fixed in 19.0.1.
I suspected there maybe also other similar cases in other packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #52633: [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3
URL: https://github.com/apache/spark/pull/52633
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #52633:
URL: https://github.com/apache/spark/pull/52633#issuecomment-3408937310
For this one, do you think we need to document somewhere this because our minimum numpy is still `1.22` and `pyarrow` is `15.0.0`.
> it was caused by a bug in 19.0.0, see https://github.com/apache/arrow/issues/45283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52633:
URL: https://github.com/apache/spark/pull/52633#issuecomment-3408999611
@zhengruifeng, I have a silly question about Python deps management - I see that many Python deps are declared without a version, or with a range version(half-bounded, e.g. `foo>=1.0` or `bar<2.0`). Silently upgrading 3rd libs may introduce breaking changes (especially for major version bumping)/bugs.
This means that if we do not specify the dependency version, or only specify the lower bound of the dependency version, PySpark may not work once a new major version of the dependency is released. This becomes a problem if users want to use older Spark versions (in practice, EOLed versions of Spark are used widely and upgrading is not timely).
I wonder if Spark can pin all Python deps in a fixed version(or at least a bounded range version, e.g. `foo>=1.0,<=2.3`), this clearly shows the versions of Spark that have been fully tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53931][INFRA][PYTHON] Fix scheduled job for numpy 2.1.3 [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #52633:
URL: https://github.com/apache/spark/pull/52633#issuecomment-3409041819
> But I personally think maybe we should use a fixed version `foo=1.2.3` or `foo>=1.0,<=2.3` in the officially released images
@zhengruifeng, that makes a lot of sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org