You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "lysnikolaou (via GitHub)" <gi...@apache.org> on 2024/08/13 12:38:48 UTC
[PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
lysnikolaou opened a new pull request, #43671:
URL: https://github.com/apache/arrow/pull/43671
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338210577
But did we actually audit the code to make sure nothing relies on the GIL? Otherwise we're just asking for trouble.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2298867524
> The fact that this was passing made me realize that the tests enable the GIL, because `cffi` is being imported. The tests now expectedly fail with a segfault.
On CI, the tests seem to be passing (and I verified that the warning about _"The global interpreter lock (GIL) has been enabled to load module '\_cffi_backend'"_ is gone)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1741947932
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
I tested this, and, unfortunately it does not work properly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750371925
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
Like @lysnikolaou I tried several variations on this and I could not make it work.
(such as `set_target_properties(Python3::Module PROPERTIES SYSTEM FALSE)`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750374929
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
The CMake docs are actually quite cryptic about this as several properties may be involved: `SYSTEM`, `NO_SYSTEM_FROM_IMPORTED` and `EXPORT_NO_SYSTEM`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1718462379
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
@kou Does this change look ok to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2298466773
Revision: 23dbf7bed719856a86620b25cf022c4e5e1a8553
Submitted crossbow builds: [ursacomputing/crossbow @ actions-b13634ff3d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b13634ff3d)
|Task|Status|
|----|------|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/10469420206/job/28992416344)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286371965
It looks like [this come from Cython](https://github.com/cython/cython/blob/1d7100d867a8b64b6f661009ec2ed45a4d8d5dba/Cython/Utility/ModuleSetupCode.c#L2399). I'll have a deeper look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1718206926
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
Okay, this was a hard one.
CMake used to add Python include directories with `-isystem`, which led to some Python-internal includes to resolve to normal 3.13 includes, instead of 3.13-free-threading, which in turn meants that `Py_GIL_DISABLED` was not set.
Setting this flag uses `-I` instead. I verified manually that the only include directories here are python-specific (Python & NumPy include directories), so this shouldn't change too much. I couldn't find how I can change this in a more granular way. If someone knows that, help would be really appreaciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2330836001
@github-actions crossbow submit -g python wheel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338173207
FWIW, the minimal-build example builds are failing (I had restarted one, and it's still failing), while I haven't seen that in the nightlies, but I also don't directly see how this PR would be causing that (can take a closer look tomorrow)
(can see if it's still failing after the rebase)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286348969
> Isn't that failure expected until #43606 is merged?
No, I don't think so. The only side-effect of **not** having `-Xfreethreading_compatible` is that the GIL is re-enabled. The error here actually says that the C extension modules were built using a non-free-threaded build, which I don't think is the case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286235185
@github-actions crossbow submit test-ubuntu-22.04-python-313-freethreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286369656
> We do pass `-DPYTHON_EXECUTABLE=/arrow-dev/bin/python3` to cmake when building pyarrow. Does the executable binary change when having a free-threaded python? (it's not that this should be `python3t` or so?)
Since the virtual environment is created using `python3.13t`, the `python` and `python3` symlinks both point to the free-threaded build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2326267545
Revision: 7f55db5771ce6cb290d04e5852ed63ab2c966986
Submitted crossbow builds: [ursacomputing/crossbow @ actions-2816971b43](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2816971b43)
|Task|Status|
|----|------|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/10681904647/job/29606534942)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338202325
@lysnikolaou I'm surprised, PyArrow is marked nogil-compatible?
```python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286342787
Isn't that failure expected until https://github.com/apache/arrow/pull/43606 is merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286220509
@github-actions crossbow submit test-ubuntu-22.04-python-313-freethreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2291320257
@pitrou Could you maybe trigger the new CI job?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2330840498
```
Unable to match any tasks for `wheel`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10716094728
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #43671:
URL: https://github.com/apache/arrow/pull/43671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338169844
Revision: dc04e0db1cc6f5182558c7b7e2a100efcb35b527
Submitted crossbow builds: [ursacomputing/crossbow @ actions-4ad45fc702](https://github.com/ursacomputing/crossbow/branches/all?query=actions-4ad45fc702)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498455/job/29876694202)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498313/job/29876693391)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498595/job/29876695779)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498265/job/29876693253)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498906/job/29876697412)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498540/job/29876694510)|
|test-conda-python-3.10-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498554/job/29876694663)|
|test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498608/job/29876695401)|
|test-conda-python-3.10-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498348/job/29876693577)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498700/job/29876696441)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498790/job/29876697073)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498597/job/29876695250)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498833/job/29876697118)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498338/job/29876693656)|
|test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498926/job/29876697520)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498249/job/29876692964)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498581/job/29876695259)|
|test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498547/job/29876694521)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498573/job/29876695038)|
|test-conda-python-3.8-pandas-1.0-numpy-1.19|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498644/job/29876695870)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498154/job/29876692811)|
|test-conda-python-3.9-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498651/job/29876696258)|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498309/job/29876693246)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/10774499035/job/29876698052)|
|test-debian-12-python-3-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498680/job/29876696475)|
|test-debian-12-python-3-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498922/job/29876697523)|
|test-fedora-39-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498732/job/29876696733)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498396/job/29876694017)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498587/job/29876695087)|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/10774498592/job/29876695687)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1742202883
##########
docker-compose.yml:
##########
@@ -1087,6 +1088,32 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading:
Review Comment:
Also, while unlikely, free-threading support is experimental in 3.13 and might not even make it to 3.14. So hardcoding 3.13 makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286303544
Hmm, we got much further this time. Build was successful, but tests failed on import with this error:
```
+ pytest -r s --pyargs pyarrow
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError
============================= test session starts ==============================
platform linux -- Python 3.13.0rc1, pytest-8.3.2, pluggy-1.5.0
rootdir: /
plugins: hypothesis-6.111.0
collected 0 items / 1 error
==================================== ERRORS ====================================
_______ ERROR collecting arrow-dev/lib/python3.13t/site-packages/pyarrow _______
usr/lib/python3.13/importlib/__init__.py:88: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
???
<frozen importlib._bootstrap>:1360: in _find_and_load
???
<frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
<frozen importlib._bootstrap>:1387: in _gcd_import
???
<frozen importlib._bootstrap>:1360: in _find_and_load
???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:935: in _load_unlocked
???
<frozen importlib._bootstrap_external>:1022: in exec_module
???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
???
arrow-dev/lib/python3.13t/site-packages/pyarrow/__init__.py:65: in <module>
import pyarrow.lib as _lib
E ImportError: Module was compiled with a non-freethreading build of Python but imported into a freethreading build.
```
That's odd, since the wheel seems to have been compiled with the free-threaded build (notice the "t" in "cp313**t**"):
```
Created wheel for pyarrow: filename=pyarrow-18.0.0.dev139-cp313-cp313t-linux_x86_64.whl size=28707111 sha256=13191763e70d010111830b51e95c8a71d582531bb471faa365d00d3daaef330f
Stored in directory: /tmp/pip-ephem-wheel-cache-h1qj9ap8/wheels/4d/35/fe/ddc412558b8012a6576905791192edd07e7e3d9c86e74f006c
Successfully built pyarrow
Installing collected packages: pyarrow
Successfully installed pyarrow-18.0.0.dev139
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2287413500
Could you create a new issue for this?
See also: https://github.com/apache/arrow/issues/43536#issuecomment-2287412282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1718206926
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
Okay, this was a hard one.
CMake used to add Python include directories with `-isystem`, which led to some Python-internal includes to resolve to normal 3.13 includes (cause `-isystem` includes are search after system directories), instead of 3.13-free-threading, which in turn meants that `Py_GIL_DISABLED` was not set.
Setting this flag uses `-I` instead. I verified manually that the only include directories here are python-specific (Python & NumPy include directories), so this shouldn't change too much. I couldn't find how I can change this in a more granular way. If someone knows that, help would be really appreaciated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2326263570
@github-actions crossbow submit test-ubuntu-22.04-python-313-freethreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338449135
Let's just let CI run a last time and merge if no further issue surfaces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750431427
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
Also, this doesn't seem to have any effect:
```console
# export PYTHON_GIL=
# python3
Python 3.13.0rc2 experimental free-threading build (main, Sep 8 2024, 23:21:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys._is_gil_enabled()
False
>>> import cffi
>>> sys._is_gil_enabled()
False
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338406133
Thank for this! I'll have a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1742022579
##########
docker-compose.yml:
##########
@@ -1087,6 +1088,32 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading:
Review Comment:
Should this be parameterized on the Python version just like `conda-python`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1719488436
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
> Can we use the `SYSTEM` target property https://cmake.org/cmake/help/latest/prop_tgt/SYSTEM.html instead of this to limit the impact?
What my change is the exact opposite though. It's signifying to not use `SYSTEM` anywhere, which I guess is what `FindPython` does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2291343142
@github-actions crossbow submit test-ubuntu-22.04-python-313-freethreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286368153
`pip` prints `Using pip 24.2 from /arrow-dev/lib/python3.13t/site-packages/pip (python 3.13)`, so that looks fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286359833
We do pass `-DPYTHON_EXECUTABLE=/arrow-dev/bin/python3` to cmake when building pyarrow. Does the executable binary change when having a free-threaded python? (it's not that this should be `python3t` or so?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338148942
Revision: dc04e0db1cc6f5182558c7b7e2a100efcb35b527
Submitted crossbow builds: [ursacomputing/crossbow @ actions-b07c305e46](https://github.com/ursacomputing/crossbow/branches/all?query=actions-b07c305e46)
|Task|Status|
|----|------|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/10774343082/job/29876177528)|
|wheel-macos-monterey-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774341995/job/29876170685)|
|wheel-macos-monterey-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342623/job/29876174618)|
|wheel-macos-monterey-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774341861/job/29876169798)|
|wheel-macos-monterey-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342200/job/29876171885)|
|wheel-macos-monterey-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774341945/job/29876170167)|
|wheel-macos-monterey-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342205/job/29876172238)|
|wheel-macos-monterey-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342121/job/29876171085)|
|wheel-macos-monterey-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342620/job/29876174553)|
|wheel-macos-monterey-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342245/job/29876172473)|
|wheel-macos-monterey-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342609/job/29876173898)|
|wheel-macos-monterey-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342936/job/29876176157)|
|wheel-macos-monterey-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342163/job/29876171803)|
|wheel-manylinux-2-28-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342805/job/29876175815)|
|wheel-manylinux-2-28-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342927/job/29876176615)|
|wheel-manylinux-2-28-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774343201/job/29876177801)|
|wheel-manylinux-2-28-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774343532/job/29876178715)|
|wheel-manylinux-2-28-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342900/job/29876176597)|
|wheel-manylinux-2-28-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774343030/job/29876177379)|
|wheel-manylinux-2-28-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342617/job/29876173900)|
|wheel-manylinux-2-28-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342823/job/29876176186)|
|wheel-manylinux-2-28-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342806/job/29876175411)|
|wheel-manylinux-2-28-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342604/job/29876174030)|
|wheel-manylinux-2-28-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342062/job/29876171151)|
|wheel-manylinux-2-28-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342064/job/29876171031)|
|wheel-manylinux-2014-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774341844/job/29876169604)|
|wheel-manylinux-2014-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342618/job/29876174453)|
|wheel-manylinux-2014-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342290/job/29876172792)|
|wheel-manylinux-2014-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774341741/job/29876169536)|
|wheel-manylinux-2014-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774343243/job/29876178047)|
|wheel-manylinux-2014-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342592/job/29876173608)|
|wheel-manylinux-2014-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342299/job/29876172856)|
|wheel-manylinux-2014-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774341915/job/29876170172)|
|wheel-manylinux-2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774343085/job/29876177080)|
|wheel-manylinux-2014-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342282/job/29876172642)|
|wheel-manylinux-2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342814/job/29876175843)|
|wheel-manylinux-2014-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342386/job/29876173217)|
|wheel-windows-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342619/job/29876174104)|
|wheel-windows-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774343167/job/29876177704)|
|wheel-windows-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342104/job/29876171190)|
|wheel-windows-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774341962/job/29876170551)|
|wheel-windows-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342809/job/29876175427)|
|wheel-windows-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10774342738/job/29876175525)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338388909
> If you could point me to such instances, that'd be really helpful, and I can work on fixing all of those.
One example is `VisitSequenceGeneric` (https://github.com/apache/arrow/blob/cd50c324882ab1419d1728e9adad20d47b185508/python/pyarrow/src/arrow/python/iterators.h#L69-L86), where the sequence's length is assumed to be constant, and where we use the borrowed ref returned by `PySequence_Fast_GET_ITEM`. Solving this one efficiently will be tricky, as CPython doesn't provide any APIs for fast and safe GIL-less access to sequences.
A similar issue lies in https://github.com/apache/arrow/blob/cd50c324882ab1419d1728e9adad20d47b185508/python/pyarrow/src/arrow/python/iterators.h#L54-L61
I haven't looked in other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338049502
Ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "cryos (via GitHub)" <gi...@apache.org>.
cryos commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750573577
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
I can take a look at this, the system behavior is more compiler defined and exposed by CMake, -I always comes before any -isystem, so if you want to be certain you get it before system headers you need to force it. That is coupled with the other behavior that -isystem silences compiler warnings from system headers. It does seem a shame to change it globally, I see this was already merged but I will see if I can spot where it falls down being more surgical about it. In general -isystem is desirable for imported targets but it gets tricky when you have multiple versions of the same package floating around!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750432818
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
Sorry for not making myself clear. No, `PYTHON_GIL` is a Python-related environment variable. Maybe [this excerpt from the Python docs](https://docs.python.org/3.13/whatsnew/3.13.html#free-threaded-cpython) would be better than me in trying to explain it?
> C-API extension modules need to be built specifically for the free-threaded build. Extensions that support running with the [GIL](https://docs.python.org/3.13/glossary.html#term-GIL) disabled should use the [Py_mod_gil](https://docs.python.org/3.13/c-api/module.html#c.Py_mod_gil) slot. Extensions using single-phase init should use [PyUnstable_Module_SetGIL()](https://docs.python.org/3.13/c-api/module.html#c.PyUnstable_Module_SetGIL) to indicate whether they support running with the GIL disabled. Importing C extensions that don’t use these mechanisms will cause the GIL to be enabled, unless the GIL was explicitly disabled with the [PYTHON_GIL](https://docs.python.org/3.13/using/cmdline.html#envvar-PYTHON_GIL) environment variable or the [-X gil=0](https://docs.python.org/3.13/using/cmdline.html#cmdoption-X) option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2290999861
Yeah, like @jorisvandenbossche said, I can't trigger CI jobs even though I have a commit in now. Could someone else please do that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2290998052
@github-actions crossbow submit test-ubuntu-22.04-python-313-freethreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286187906
Revision: 1cb315b37998fd67541997da8c9f616aff82d64f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-5e60b16dcf](https://github.com/ursacomputing/crossbow/branches/all?query=actions-5e60b16dcf)
|Task|Status|
|----|------|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/10370065249/job/28707149016)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286239951
Revision: d16c04c759bff98b12c86ab2d4a95070ac72ed18
Submitted crossbow builds: [ursacomputing/crossbow @ actions-37df7dbfe0](https://github.com/ursacomputing/crossbow/branches/all?query=actions-37df7dbfe0)
|Task|Status|
|----|------|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/10370550281/job/28708691315)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286222374
```
Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10370420938
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2330864485
Revision: 3c453c556bca1611bbac3fa3f97ccc937f533200
Submitted crossbow builds: [ursacomputing/crossbow @ actions-01fc0ace35](https://github.com/ursacomputing/crossbow/branches/all?query=actions-01fc0ace35)
|Task|Status|
|----|------|
|example-python-minimal-build-fedora-conda|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275534/job/29713420208)|
|example-python-minimal-build-ubuntu-venv|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276251/job/29713425468)|
|python-sdist|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276693/job/29713427323)|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276580/job/29713427057)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276568/job/29713427034)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276198/job/29713428346)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276732/job/29713428622)|
|test-conda-python-3.10-pandas-latest-numpy-1.26|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276554/job/29713427053)|
|test-conda-python-3.10-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276114/job/29713425206)|
|test-conda-python-3.10-pandas-nightly-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276439/job/29713429799)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275788/job/29713422200)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275703/job/29713421576)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276433/job/29713426803)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276202/job/29713426524)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276907/job/29713466893)|
|test-conda-python-3.11-pandas-upstream_devel-numpy-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276639/job/29713427086)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275668/job/29713420968)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276229/job/29713428384)|
|test-conda-python-3.12-cpython-debug|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275965/job/29713422607)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276678/job/29713430051)|
|test-conda-python-3.8-pandas-1.0-numpy-1.19|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275554/job/29713420251)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276887/job/29713430448)|
|test-conda-python-3.9-pandas-latest-numpy-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275945/job/29713422668)|
|test-conda-python-emscripten|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275865/job/29713422416)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276461/job/29713426817)|
|test-debian-12-python-3-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276304/job/29713425673)|
|test-debian-12-python-3-i386|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275577/job/29713420237)|
|test-fedora-39-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276446/job/29713426849)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276893/job/29713430408)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276211/job/29713428353)|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276194/job/29713425437)|
|wheel-macos-monterey-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276080/job/29713425172)|
|wheel-macos-monterey-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276370/job/29713429578)|
|wheel-macos-monterey-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275912/job/29713422420)|
|wheel-macos-monterey-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275807/job/29713422397)|
|wheel-macos-monterey-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276651/job/29713429858)|
|wheel-macos-monterey-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276291/job/29713428410)|
|wheel-macos-monterey-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275988/job/29713423487)|
|wheel-macos-monterey-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275755/job/29713422160)|
|wheel-macos-monterey-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275704/job/29713421557)|
|wheel-macos-monterey-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275764/job/29713422199)|
|wheel-macos-monterey-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276839/job/29713430395)|
|wheel-macos-monterey-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276926/job/29713466932)|
|wheel-manylinux-2-28-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276416/job/29713429608)|
|wheel-manylinux-2-28-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276699/job/29713430098)|
|wheel-manylinux-2-28-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276379/job/29713426796)|
|wheel-manylinux-2-28-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275982/job/29713423484)|
|wheel-manylinux-2-28-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276804/job/29713430168)|
|wheel-manylinux-2-28-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276326/job/29713426514)|
|wheel-manylinux-2-28-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276039/job/29713425181)|
|wheel-manylinux-2-28-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276952/job/29713467150)|
|wheel-manylinux-2-28-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275802/job/29713422165)|
|wheel-manylinux-2-28-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275605/job/29713420629)|
|wheel-manylinux-2-28-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276069/job/29713423519)|
|wheel-manylinux-2-28-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276154/job/29713424196)|
|wheel-manylinux-2014-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276195/job/29713425429)|
|wheel-manylinux-2014-cp310-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276642/job/29713427251)|
|wheel-manylinux-2014-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275600/job/29713420437)|
|wheel-manylinux-2014-cp311-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276115/job/29713425195)|
|wheel-manylinux-2014-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276818/job/29713430326)|
|wheel-manylinux-2014-cp312-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275695/job/29713421529)|
|wheel-manylinux-2014-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275930/job/29713422452)|
|wheel-manylinux-2014-cp313-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275637/job/29713421014)|
|wheel-manylinux-2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276914/job/29713466910)|
|wheel-manylinux-2014-cp38-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276701/job/29713430049)|
|wheel-manylinux-2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276103/job/29713424180)|
|wheel-manylinux-2014-cp39-arm64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276212/job/29713425453)|
|wheel-windows-cp310-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276686/job/29713427329)|
|wheel-windows-cp311-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276703/job/29713427277)|
|wheel-windows-cp312-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716275563/job/29713420244)|
|wheel-windows-cp313-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276025/job/29713423490)|
|wheel-windows-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716276477/job/29713429878)|
|wheel-windows-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions/runs/10716277022/job/29713467197)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2339343842
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit d28e542ef4dafcbb37abc5501154ee45b1b16b6f.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/29904284629) has more details. It also includes information about 39 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750324880
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
But where is this env var taken into account?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1742038338
##########
docker-compose.yml:
##########
@@ -1087,6 +1088,32 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading:
Review Comment:
Eventually yes, but was thinking that we could also handle that when there are actually multiple python versions that could be used here (but also fine to already do that, of course)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2298272077
@github-actions crossbow submit test-ubuntu-22.04-python-313-freethreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2291346990
Revision: 4d4a69c2e6d634dbdc8daf553c552ed6b07c06fa
Submitted crossbow builds: [ursacomputing/crossbow @ actions-7fa0aa9f6c](https://github.com/ursacomputing/crossbow/branches/all?query=actions-7fa0aa9f6c)
|Task|Status|
|----|------|
|test-ubuntu-22.04-python-313-freethreading|[](https://github.com/ursacomputing/crossbow/actions/runs/10405198547/job/28815414020)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1718111983
##########
docker-compose.yml:
##########
@@ -150,6 +150,7 @@ x-hierarchy:
- ubuntu-lint
- ubuntu-python
- ubuntu-python-sdist-test
+ - ubuntu-python-313-freethreading-test
Review Comment:
```suggestion
- ubuntu-python-313-freethreading
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750426545
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
When running the test suite, `cffi` is being imported. Because `cffi` has not declared support for free-threading yet, the GIL will be enabled, resulting in the tests running with the GIL enabled. If `PYTHON_GIL` has been set to `0`, the enabling of the GIL is prevented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338161389
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338207786
> @lysnikolaou I'm surprised, PyArrow is marked nogil-compatible?
Yes, this was done in https://github.com/apache/arrow/pull/43606.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2290998767
```
Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/10402069621
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1719173490
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
Can we use the `SYSTEM` target property https://cmake.org/cmake/help/latest/prop_tgt/SYSTEM.html instead of this to limit the impact?
```diff
diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
index 5d5eeaf815..c19820c074 100644
--- a/python/CMakeLists.txt
+++ b/python/CMakeLists.txt
@@ -258,6 +258,8 @@ set(EXECUTABLE_OUTPUT_PATH "${BUILD_OUTPUT_ROOT_DIRECTORY}")
find_package(Python3Alt REQUIRED)
message(STATUS "Found NumPy version: ${Python3_NumPy_VERSION}")
message(STATUS "NumPy include dir: ${NUMPY_INCLUDE_DIRS}")
+# TODO: Describe why we need this
+set_target_properties(Python3::Python PROPERTIES SYSTEM FALSE)
include(UseCython)
message(STATUS "Found Cython version: ${CYTHON_VERSION}")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338327266
> Regarding the C/C++ code, I had a look around the codebase, trying to manually search for global state, but I didn't find any instances that might create problems.
Besides explicitly global state, there may also be cases of relying on the GIL to ensure that e.g. a Python dict/list isn't mutated, or a borrowed reference remains valid.
> As a note, being agressive with declaring support for free-threading is probably the way to go, since bugs will always be there, and they will be much harder to surface if widely-used packages do not declare support.
That is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750419748
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
@kou We'll have to live with this, unless you want to diagnose the issue yourself. Understanding CMake's intricacies is no fun.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1716117766
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # PYARROW_VERSION: The test target pyarrow version such as "3.0.0"
+ # UBUNTU: 20.04, 22.04
+ image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
+ build:
+ context: .
+ dockerfile: ci/docker/linux-apt-python-313-freethreading-test.dockerfile
+ cache_from:
+ - ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
+ args:
+ base: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp
+ shm_size: *shm-size
+ environment:
+ <<: [*common, *ccache]
+ # Bundled build of OpenTelemetry needs a git client
+ ARROW_WITH_OPENTELEMETRY: "OFF"
+ PYARROW_VERSION: ${PYARROW_VERSION:-}
+ volumes: *ubuntu-volumes
+ command: >
+ /bin/bash -c "
+ /arrow/ci/scripts/cpp_build.sh /arrow /build &&
+ /arrow/ci/scripts/python_build.sh /arrow /build &&
+ /arrow/ci/scripts/python_test.sh /arrow"
Review Comment:
```suggestion
command: *python-command
```
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
Review Comment:
Could you remove `-test` suffix like we did for `ubuntu-python`?
```suggestion
ubuntu-python-313-freethreading:
```
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # PYARROW_VERSION: The test target pyarrow version such as "3.0.0"
+ # UBUNTU: 20.04, 22.04
+ image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
+ build:
+ context: .
+ dockerfile: ci/docker/linux-apt-python-313-freethreading-test.dockerfile
Review Comment:
```suggestion
dockerfile: ci/docker/linux-apt-python-313-freethreading.dockerfile
```
##########
ci/docker/linux-apt-python-313-freethreading-test.dockerfile:
##########
Review Comment:
Could you remove `-test` from filename?
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # PYARROW_VERSION: The test target pyarrow version such as "3.0.0"
+ # UBUNTU: 20.04, 22.04
+ image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
+ build:
+ context: .
+ dockerfile: ci/docker/linux-apt-python-313-freethreading-test.dockerfile
+ cache_from:
+ - ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
Review Comment:
```suggestion
- ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading
```
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # PYARROW_VERSION: The test target pyarrow version such as "3.0.0"
+ # UBUNTU: 20.04, 22.04
+ image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
+ build:
+ context: .
+ dockerfile: ci/docker/linux-apt-python-313-freethreading-test.dockerfile
+ cache_from:
+ - ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
+ args:
+ base: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cpp
+ shm_size: *shm-size
+ environment:
+ <<: [*common, *ccache]
+ # Bundled build of OpenTelemetry needs a git client
+ ARROW_WITH_OPENTELEMETRY: "OFF"
+ PYARROW_VERSION: ${PYARROW_VERSION:-}
Review Comment:
Do we need this?
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # PYARROW_VERSION: The test target pyarrow version such as "3.0.0"
+ # UBUNTU: 20.04, 22.04
+ image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading-test
Review Comment:
```suggestion
image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-python-313-freethreading
```
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # PYARROW_VERSION: The test target pyarrow version such as "3.0.0"
+ # UBUNTU: 20.04, 22.04
Review Comment:
```suggestion
# UBUNTU: 20.04, 22.04, 24.04
```
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
+ # Parameters:
+ # ARCH: amd64, arm64v8, ...
+ # PYARROW_VERSION: The test target pyarrow version such as "3.0.0"
Review Comment:
Do we need this?
##########
docker-compose.yml:
##########
@@ -1086,6 +1087,38 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading-test:
+ # Usage:
+ # docker-compose build ubuntu-cpp
+ # docker-compose build ubuntu-python-313-freethreading-test
+ # docker-compose run --rm ubuntu-python-313-freethreading-test
Review Comment:
```suggestion
# docker-compose build ubuntu-python-313-freethreading
# docker-compose run --rm ubuntu-python-313-freethreading
```
##########
dev/tasks/tasks.yml:
##########
@@ -1254,6 +1254,14 @@ tasks:
PYTHON: "3.10"
image: conda-python-cython2
+ test-ubuntu-22.04-python-313-freethreading:
+ ci: github
+ template: docker-tests/github.linux.yml
+ params:
+ env:
+ UBUNTU: 22.04
+ image: ubuntu-python-313-freethreading-test
Review Comment:
```suggestion
image: ubuntu-python-313-freethreading
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2326252279
This can be merged IMO. I got one segfault that was happening in `test_cffi`, but I cannot reliably reproduce any more even after running the tests with `pytest-repeat` and `--count=100`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1731317705
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
> > Can we use the `SYSTEM` target property https://cmake.org/cmake/help/latest/prop_tgt/SYSTEM.html instead of this to limit the impact?
>
> What my change is the exact opposite though. It's signifying to not use `SYSTEM` anywhere, which I guess is what `FindPython` does.
Unless I am mistaken @kou is suggesting, to set `SYSTEM` to `FALSE` for Python but leave it as is for the rest of dependencies instead of changing it globally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2330858223
@github-actions crossbow submit -g python -g wheel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338141378
@github-actions crossbow submit -g wheel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750269212
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
Is the `PYTHON_GIL` environment variable used somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338201045
> FWIW, the minimal-build example builds are failing (I had restarted one, and it's still failing), while I haven't seen that in the nightlies, but I also don't directly see how this PR would be causing that (can take a closer look tomorrow)
Yes, it's unrelated. The problem is that the fork from which the PR is submitted doesn't have any git tags:
https://github.com/ursacomputing/crossbow/actions/runs/10774498455/job/29876694202#step:3:3401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750322842
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
It's used so that the test suite runs with the GIL disabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338301794
> But did we actually audit the code to make sure nothing relies on the GIL? Otherwise we're just asking for trouble.
Since a big chuck of PyArrow relies on Cython, that means that a lot of the free-threading heavy-lifting will be performed by Cython.
Regarding the C/C++ code, I had a look around the codebase, trying to manually search for global state, but I didn't find any instances that might create problems. I also talked a bit with @jorisvandenbossche about that during a call, and adding threading-related tests was something he wanted to do as well.
As a note, being agressive with declaring support for free-threading is probably the way to go, since bugs will always be there, and they will be much harder to surface if widely-used packages do no declare support. Remember that the GIL is enabled for the whole process, even if just one package hasn't declared support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2338339411
> there may also be cases of relying on the GIL to ensure that e.g. a Python dict/list isn't mutated
If you could point me to such instances, that'd be really helpful, and I can work on fixing all of those.
> or a borrowed reference remains valid
We got rid of borrowed references APIs in #43540.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750427650
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
So `PYTHON_GIL` is a cffi-specific environment variable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750420778
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
Ping @lysnikolaou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1750433570
##########
ci/docker/linux-apt-python-313-freethreading.dockerfile:
##########
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+ARG base
+FROM ${base}
+
+RUN apt install -y -q --no-install-recommends software-properties-common gpg-agent
+RUN add-apt-repository -y ppa:deadsnakes/ppa && apt update
+RUN apt install -y -q --no-install-recommends python3.13-dev python3.13-nogil python3.13-venv
+
+COPY python/requirements-build.txt \
+ python/requirements-test.txt \
+ /arrow/python/
+
+ENV ARROW_PYTHON_VENV /arrow-dev
+RUN python3.13t -m venv ${ARROW_PYTHON_VENV}
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install -U pip setuptools wheel
+RUN ${ARROW_PYTHON_VENV}/bin/python -m pip install \
+ --pre \
+ --prefer-binary \
+ --extra-index-url "https://pypi.anaconda.org/scientific-python-nightly-wheels/simple" \
+ -r arrow/python/requirements-build.txt \
+ -r arrow/python/requirements-test.txt
+
+ENV ARROW_ACERO=ON \
+ ARROW_BUILD_STATIC=OFF \
+ ARROW_BUILD_TESTS=OFF \
+ ARROW_BUILD_UTILITIES=OFF \
+ ARROW_COMPUTE=ON \
+ ARROW_CSV=ON \
+ ARROW_DATASET=ON \
+ ARROW_FILESYSTEM=ON \
+ ARROW_GDB=ON \
+ ARROW_HDFS=ON \
+ ARROW_JSON=ON \
+ ARROW_USE_GLOG=OFF \
+ PYTHON_GIL=0
Review Comment:
Ah, I see it's documented here: https://docs.python.org/3.14/using/cmdline.html#envvar-PYTHON_GIL
Google isn't very good at finding it...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1742063441
##########
docker-compose.yml:
##########
@@ -1087,6 +1088,32 @@ services:
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
/arrow/ci/scripts/python_sdist_test.sh /arrow"
+ ############################ Python free-threading ##########################
+
+ ubuntu-python-313-freethreading:
Review Comment:
That might influence on Docker image name and caching, though that would admittedly be a one-off annoyance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "lysnikolaou (via GitHub)" <gi...@apache.org>.
lysnikolaou commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2291680520
The fact that this was passing made me realize that the tests enable the GIL, because `cffi` is being imported (see https://github.com/ursacomputing/crossbow/actions/runs/10405198547/job/28815414020#step:7:7070). The tests now expectedly fail with a segfault. I'm working on figuring what's happening there, but that'll probably be in a couple of weeks, when I'm back from vacation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #43671:
URL: https://github.com/apache/arrow/pull/43671#issuecomment-2286183259
@github-actions crossbow submit test-ubuntu-22.04-python-313-freethreading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43536: [Python][CI] Add a Crossbow job with the free-threaded build [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43671:
URL: https://github.com/apache/arrow/pull/43671#discussion_r1751254229
##########
python/CMakeLists.txt:
##########
@@ -21,6 +21,8 @@
cmake_minimum_required(VERSION 3.16)
project(pyarrow)
+set(CMAKE_NO_SYSTEM_FROM_IMPORTED ON)
Review Comment:
OK. I'll try it later by myself too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org