You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "alippai (via GitHub)" <gi...@apache.org> on 2025/12/12 14:11:33 UTC
[I] Drop numpy 1.x support [arrow]
alippai opened a new issue, #48473:
URL: https://github.com/apache/arrow/issues/48473
### Describe the enhancement requested
Numpy 1.26 (last of the 1.x version) has reached end of life in September 2025
### Component(s)
Python, Packaging, Release, Documentation, Integration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5020733342
Thanks for linking the issue @jorisvandenbossche , I was not aware of it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x and pandas 2.x [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-3646757185
@jorisvandenbossche @rok @AlenkaF Does this sound reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x and pandas 2.x [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-3646831786
Yes, definitely, it is time to drop pandas 1.x support
For numpy 1.26 we might want wait a bit longer, although nowadays I think most projects have updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5020356816
> specifically for numpy variable-width string support.
Aren't these two different things? One dropping support for numpy < 2.x and another one to improve support around numpy variable-width string. We should create an issue for the second one, I don't think it is tracked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #48473: [Python] Require numpy 2.x
URL: https://github.com/apache/arrow/issues/48473
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5205553237
> 2. it would be best to remove numpy<2.x to make this cleaner
Do you mean "cleaner" code-wise? i.e. that it is easier to use numpy 2.0 C API things when requiring numpy 2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x and pandas 2.x [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-3674031221
Recommendation from NumPy developer to only drop 1.26 the earliest mid 2026: https://github.com/scientific-python/specs/issues/386#issuecomment-3619804433 (and as Arrow, we might even want to be a bit more conservative)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x and pandas 2.x [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-3646793532
Last 1.x was [more than 24 months ago](https://endoflife.date/numpy), so it sounds ok. I would appreciate @jorisvandenbossche 's opinion though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-4175969724
We should remove this fix from the musllinux wheels dockerfile when bumping numpy:
https://github.com/apache/arrow/pull/49639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "ngoldbaum (via GitHub)" <gi...@apache.org>.
ngoldbaum commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5208783651
> Do you mean "cleaner" code-wise? i.e. that it is easier to use numpy 2.0 C API things when requiring numpy 2?
I looked at this today and it turns out it should be possible to get StringDType support working while still allowing NumPy 1.26 at runtime. The extra complexity is that to work with StringDType from C++, you'll need to use the [NpyString C API](https://numpy.org/devdocs/reference/c-api/strings.html#npystring-api), which is only available on NumPy 2.0 and newer.
If I have NumPy 1.26 installed and I try to import an extension that declares via `NPY_TARGET_VERSION=NPY_2_0_API_VERSION` that it needs to use the NumPy 2.0 C API, I'll see an ImportError at runtime inside of NumPy's C `import_array()` hook.
However, only one extension module needs to do this: all of PyArrow extension modules don't need to use the new NumPy C API version. Instead, you could put the things that need the NumPy 2.0 C API into their own extension module that needs NumPy 2.0 or newer at runtime. Then, you could only import that extension after doing the "regular" numpy import and checking NumPy's version to see if NumPy 2.0 is available at runtime.
So, not *too bad*, but definitely more complicated than #48473. I pinged the arrow-dev mailing list to see if anyone would be annoyed with dropping runtime support for older NumPy versions.
I double-checked and arrow's wheel builds already have a `numpy>=2.0.0` bound in the requirements files and Arrow's CI already explicitly tests against NumPy 1.X at runtime to make sure NumPy's simultaneous backward and forward compatibility trick works in-practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5373012569
Issue resolved by pull request 50769
https://github.com/apache/arrow/pull/50769
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x and pandas 2.x [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-3674144880
> Recommendation from NumPy developer to only drop 1.26 the earliest mid 2026: [scientific-python/specs#386 (comment)](https://github.com/scientific-python/specs/issues/386#issuecomment-3619804433) (and as Arrow, we might even want to be a bit more conservative)
I see. Well, it depends what the cost is for us of still supporting 1.26, I'd days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-4356217213
Do you think Arrow 25 is a good target for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5021425255
The discussion at europython was: 1. we should have zero-copy between arrow and numpy string types, 2. it would be best to remove numpy<2.x to make this cleaner. Hence we put the comment here as it's the prerequisite for the issue Joris linked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5212372350
I made the working PR with the old API a few months ago and there was some pushback that it's confusing, hard to read.
That can be revived if there is interest, but I believe the NumPy 2.0 should be a solid choice for Arrow 26.
The extension module trick is cool. Alternatively you can infer that python 3.13-3.14 is Numpy 2.0 for sure (and skip new numpy api support for python 3.10-3.12).
https://github.com/scientific-python/specs/issues/386#issuecomment-5205781428 here is a new update on the SPEC0 confirming the change.
While we officially don't follow SPEC0, they already dropped NumPy 2.0 and they are dropping NumPy 2.1 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5020637332
> another one to improve support around numpy variable-width string. We should create an issue for the second one, I don't think it is tracked.
We have https://github.com/apache/arrow/issues/42018 for supporting basic conversion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Require numpy 2.x [arrow]
Posted by "tadeja (via GitHub)" <gi...@apache.org>.
tadeja commented on issue #48473:
URL: https://github.com/apache/arrow/issues/48473#issuecomment-5010644304
fyi @ngoldbaum was talking with me about this at the europython sprints, specifically for numpy variable-width string support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org