You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ti...@apache.org on 2026/05/29 20:53:24 UTC
(datafusion-python) branch main updated: feat: create free-threaded python wheels (#1553)
This is an automated email from the ASF dual-hosted git repository.
timsaucer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-python.git
The following commit(s) were added to refs/heads/main by this push:
new 3d4c56c0 feat: create free-threaded python wheels (#1553)
3d4c56c0 is described below
commit 3d4c56c0757ddc0372ced03ffa97a13ea50d1bd8
Author: Tim Saucer <ti...@gmail.com>
AuthorDate: Fri May 29 16:53:19 2026 -0400
feat: create free-threaded python wheels (#1553)
* Initial commit for free threaded python support
* ci: use uvx to run maturin in native wheel builds
The free-threaded matrix entries skip `uv sync` to avoid resolving
project dependencies against cp313t/cp314t (many dev deps lack
free-threaded wheels), so `uv run --no-project maturin` failed on
macOS/Windows with "Failed to spawn: `maturin`". Switch to
`uvx maturin@1.8.1`, which runs maturin in an isolated tool env
independent of the project venv and matches the pin used by
maturin-action for manylinux builds.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* ci: resolve free-threaded interpreter path explicitly on Windows
maturin's `--interpreter python3.14t` fails on Windows because the
free-threaded build ships as plain `python.exe` (no `tN` suffix). Look
up `sys.executable` of the python on PATH (which actions/setup-python
prepends with the free-threaded install), assert
`Py_GIL_DISABLED == 1` so a misconfigured PATH can't silently build a
GIL wheel, and normalize backslashes to forward slashes so the path
survives re-expansion in the downstream `run:` line.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* build: enable PyO3 generate-import-lib for Windows free-threaded wheels
Windows free-threaded Python does not expose `abiflags` in sysconfig,
so PyO3's default Windows linkage path fails with "A python 3
interpreter on Windows does not define abiflags in its sysconfig ಠ_ಠ"
when building cp31Xt wheels. Enabling the `generate-import-lib` PyO3
feature switches Windows builds to a generated import library
(provided by the `python3-dll-a` crate) that does not depend on a
fully populated sysconfig. It is a no-op on macOS and Linux and is
compatible with the existing `abi3` feature.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* ci: bump maturin to 1.13.3 for Windows free-threaded support
maturin 1.8.1 errors out on Windows free-threaded interpreters with
"A python 3 interpreter on Windows does not define abiflags in its
sysconfig" even when given a valid `python.exe`. Newer maturin
releases handle the missing abiflags gracefully for cp31Xt builds.
Bump both the `uvx maturin@` pin used for native macOS/Windows wheels
and the `maturin-version` passed to PyO3/maturin-action for the
manylinux containers.
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* ci: standardize wheel build job names as "<OS> <arch> (<tag>)"
The mac/Windows matrix shared a single name template that prepended
"macOS arm64 & Windows" to every entry, which got truncated in the
GitHub UI sidebar and made it hard to tell macOS and Windows runs
apart. Rename all wheel build jobs to the same pattern so the OS,
architecture, and python tag are visible at a glance:
- Linux x86_64 / arm64
- macOS arm64 / x86_64
- Windows x86_64
Co-Authored-By: Claude Opus 4.7 (1M context) <no...@anthropic.com>
* taplo fmt
* build: move pygithub to release group to fix free-threaded wheel builds
pygithub pulls in cryptography via pyjwt[crypto]. cryptography 44.0.0
ships only abi3 wheels, which free-threaded interpreters cannot use, so
uv builds it from sdist; its bundled PyO3 0.23.2 caps at Python 3.13 and
fails on 3.14t. pygithub is only used by the manual release changelog
script, so move it out of the dev group into a new release group.
'uv sync --dev' (used by CI test jobs) no longer drags in cryptography.
* ci: pin uv venv to setup-python interpreter for free-threaded jobs
Passing a bare version like '3.13t' to 'uv venv --python' let uv fall
back to a different system interpreter (3.12), creating a venv whose ABI
did not match the downloaded cp313t wheel and failing the install. Use
the python-path output from setup-python so the venv uses exactly the
interpreter that was set up.
* taplo fmt
* ci: set UV_PYTHON so uv sync keeps the free-threaded interpreter
Pinning only 'uv venv --python' was not enough: 'uv sync' ignores the
existing .venv, runs its own interpreter discovery, and recreated the
venv with the system 3.12, again mismatching the cp313t wheel. Set
UV_PYTHON to the setup-python interpreter for the install and test
steps so every uv command (venv, sync, pip, run) uses it.
* ci: run tests from the .venv, not the bare setup-python interpreter
Setting UV_PYTHON on the test step pointed 'uv run --no-project pytest'
at the setup-python interpreter, which has no pytest installed, causing
'Failed to spawn: pytest'. UV_PYTHON is only needed in the install step
to build the .venv with the right interpreter; the test step must use
that .venv. Drop UV_PYTHON from the test step.
Co-Authored-By: Claude <no...@anthropic.com>
* ci: install datafusion wheel into the activated .venv
Setting UV_PYTHON as a step env split the install across two
environments: 'uv sync' populated .venv while 'uv pip install' targeted
the bare setup-python interpreter, so the datafusion wheel never landed
in .venv and 'import datafusion' failed under pytest. Pin the
interpreter at 'uv venv --python', activate the venv, and pass --active
to 'uv sync' so sync and pip install both target the same .venv.
Co-Authored-By: Claude <no...@anthropic.com>
* ci: point uv at the venv interpreter by path for free-threaded jobs
Activating the venv and passing --active still let 'uv sync' run its own
interpreter discovery, which skips free-threaded builds and re-picked the
system 3.12, recreating .venv and breaking the cp313t/cp314t wheel
install. Pass the venv's own interpreter (.venv/bin/python) explicitly to
'uv sync', 'uv pip install', and 'uv run' so every step stays in the
free-threaded environment created by 'uv venv'.
Co-Authored-By: Claude <no...@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <no...@anthropic.com>
---
.github/actions/build-wheel/action.yml | 110 ++++++++++++++++++++++++++++
.github/workflows/build.yml | 128 ++++++++++++++++++---------------
.github/workflows/test.yml | 62 ++++++++++------
Cargo.lock | 10 +++
crates/core/Cargo.toml | 9 ++-
dev/release/README.md | 1 +
pyproject.toml | 6 +-
uv.lock | 6 +-
8 files changed, 247 insertions(+), 85 deletions(-)
diff --git a/.github/actions/build-wheel/action.yml b/.github/actions/build-wheel/action.yml
new file mode 100644
index 00000000..25f75d1f
--- /dev/null
+++ b/.github/actions/build-wheel/action.yml
@@ -0,0 +1,110 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Composite action that builds a datafusion-python wheel with maturin.
+# Centralises the abi3-vs-free-threaded argument logic so platform jobs
+# stay short and changes to wheel-build flags happen in one place.
+
+name: "Build wheel"
+description: "Build datafusion-python wheel with maturin (abi3 or free-threaded)"
+
+inputs:
+ target:
+ description: "Rust target triple (e.g. x86_64-unknown-linux-gnu). Required when manylinux is set; ignored for native builds."
+ required: false
+ default: ""
+ python-tag:
+ description: "abi3 (covers 3.10..3.14 GIL builds) or a free-threaded interpreter such as 3.13t / 3.14t"
+ required: true
+ build-mode:
+ description: "release or debug"
+ required: true
+ features:
+ description: "Comma-separated extra features (in addition to those implied by the python-tag)"
+ required: false
+ default: "substrait"
+ manylinux:
+ description: "manylinux tag for maturin-action (e.g. 2_28). Leave empty to use uv-run maturin natively."
+ required: false
+ default: ""
+ out-dir:
+ description: "Output directory for built wheels"
+ required: false
+ default: "dist"
+
+outputs:
+ args:
+ description: "Computed maturin args (for debugging)"
+ value: ${{ steps.args.outputs.args }}
+
+runs:
+ using: "composite"
+ steps:
+ - name: Compute maturin args
+ id: args
+ shell: bash
+ run: |
+ set -euo pipefail
+ FEATURES="${{ inputs.features }}"
+ TAG="${{ inputs.python-tag }}"
+ if [ "$TAG" = "abi3" ]; then
+ # Default features include the `abi3` cargo feature.
+ # One wheel covers Python 3.10..3.14 (GIL builds only).
+ BUILD_ARGS="--features ${FEATURES}"
+ else
+ # Free-threaded build: disable abi3, force mimalloc back in, pin interpreter.
+ if [ "${RUNNER_OS:-}" = "Windows" ]; then
+ # Windows free-threaded builds ship as `python.exe` (no `tN`
+ # suffix). Resolve sys.executable so the path is independent of
+ # PATH ordering, and assert the interpreter is actually
+ # free-threaded before we hand the wheel off.
+ INTERP=$(python -c 'import sys; print(sys.executable)')
+ python -c "import sysconfig, sys; \
+ v = sysconfig.get_config_var('Py_GIL_DISABLED'); \
+ sys.exit(0 if v == 1 else f'expected free-threaded interpreter, got Py_GIL_DISABLED={v!r} at {sys.executable}')"
+ # Backslashes in BUILD_ARGS would be parsed as escapes when the
+ # output is re-expanded in the next step; use forward slashes
+ # (maturin/Rust accept them on Windows).
+ INTERP="${INTERP//\\//}"
+ else
+ INTERP="python${TAG}"
+ fi
+ BUILD_ARGS="--no-default-features --features mimalloc,${FEATURES} --interpreter ${INTERP}"
+ fi
+ if [ "${{ inputs.build-mode }}" = "release" ]; then
+ BUILD_ARGS="--release --strip ${BUILD_ARGS}"
+ fi
+ BUILD_ARGS="${BUILD_ARGS} --out ${{ inputs.out-dir }}"
+ echo "args=${BUILD_ARGS}" >> "$GITHUB_OUTPUT"
+ echo "maturin args: ${BUILD_ARGS}"
+
+ - name: Build via maturin-action (manylinux container)
+ if: inputs.manylinux != ''
+ uses: PyO3/maturin-action@v1
+ with:
+ target: ${{ inputs.target }}
+ manylinux: ${{ inputs.manylinux }}
+ maturin-version: "1.13.3"
+ args: ${{ steps.args.outputs.args }}
+ rustup-components: rust-std
+
+ - name: Build via native maturin
+ if: inputs.manylinux == ''
+ shell: bash
+ # Use `uvx` so maturin is available even when `uv sync` was skipped
+ # (free-threaded matrix entries don't pre-populate the project venv).
+ run: uvx maturin@1.13.3 build ${{ steps.args.outputs.args }}
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 37a9dba0..593a343e 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -135,8 +135,12 @@ jobs:
# ============================================
build-manylinux-x86_64:
needs: [generate-license, lint-rust, lint-python]
- name: ManyLinux x86_64
+ name: Linux x86_64 (${{ matrix.python-tag }})
runs-on: ubuntu-latest
+ strategy:
+ fail-fast: false
+ matrix:
+ python-tag: ["abi3", "3.13t", "3.14t"]
steps:
- uses: actions/checkout@v6
@@ -153,7 +157,7 @@ jobs:
- name: Cache Cargo
uses: Swatinem/rust-cache@v2
with:
- key: ${{ inputs.build_mode }}
+ key: ${{ inputs.build_mode }}-${{ matrix.python-tag }}
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b
with:
@@ -172,25 +176,18 @@ jobs:
free -h
swapon --show
- - name: Build (release mode)
- uses: PyO3/maturin-action@v1
- if: inputs.build_mode == 'release'
- with:
- target: x86_64-unknown-linux-gnu
- manylinux: "2_28"
- args: --release --strip --features protoc,substrait --out dist
- rustup-components: rust-std
-
- - name: Build (debug mode)
- uses: PyO3/maturin-action@v1
- if: inputs.build_mode == 'debug'
+ - name: Build wheel
+ uses: ./.github/actions/build-wheel
with:
target: x86_64-unknown-linux-gnu
+ python-tag: ${{ matrix.python-tag }}
+ build-mode: ${{ inputs.build_mode }}
+ features: "protoc,substrait"
manylinux: "2_28"
- args: --features protoc,substrait --out dist
- rustup-components: rust-std
+ # FFI test wheel only needs to be built once per platform; gate to abi3.
- name: Build FFI test library
+ if: matrix.python-tag == 'abi3'
uses: PyO3/maturin-action@v1
with:
target: x86_64-unknown-linux-gnu
@@ -202,10 +199,11 @@ jobs:
- name: Archive wheels
uses: actions/upload-artifact@v7
with:
- name: dist-manylinux-x86_64
+ name: dist-manylinux-x86_64-${{ matrix.python-tag }}
path: dist/*
- name: Archive FFI test wheel
+ if: matrix.python-tag == 'abi3'
uses: actions/upload-artifact@v7
with:
name: test-ffi-manylinux-x86_64
@@ -216,8 +214,12 @@ jobs:
# ============================================
build-manylinux-aarch64:
needs: [generate-license, lint-rust, lint-python]
- name: ManyLinux arm64
+ name: Linux arm64 (${{ matrix.python-tag }})
runs-on: ubuntu-24.04-arm
+ strategy:
+ fail-fast: false
+ matrix:
+ python-tag: ["abi3", "3.13t", "3.14t"]
steps:
- uses: actions/checkout@v6
@@ -234,7 +236,7 @@ jobs:
- name: Cache Cargo
uses: Swatinem/rust-cache@v2
with:
- key: ${{ inputs.build_mode }}
+ key: ${{ inputs.build_mode }}-${{ matrix.python-tag }}
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b
with:
@@ -253,29 +255,20 @@ jobs:
free -h
swapon --show
- - name: Build (release mode)
- uses: PyO3/maturin-action@v1
- if: inputs.build_mode == 'release'
- with:
- target: aarch64-unknown-linux-gnu
- manylinux: "2_28"
- args: --release --strip --features protoc,substrait --out dist
- rustup-components: rust-std
-
- - name: Build (debug mode)
- uses: PyO3/maturin-action@v1
- if: inputs.build_mode == 'debug'
+ - name: Build wheel
+ uses: ./.github/actions/build-wheel
with:
target: aarch64-unknown-linux-gnu
+ python-tag: ${{ matrix.python-tag }}
+ build-mode: ${{ inputs.build_mode }}
+ features: "protoc,substrait"
manylinux: "2_28"
- args: --features protoc,substrait --out dist
- rustup-components: rust-std
- name: Archive wheels
uses: actions/upload-artifact@v7
if: inputs.build_mode == 'release'
with:
- name: dist-manylinux-aarch64
+ name: dist-manylinux-aarch64-${{ matrix.python-tag }}
path: dist/*
# ============================================
@@ -283,13 +276,13 @@ jobs:
# ============================================
build-python-mac-win:
needs: [generate-license, lint-rust, lint-python]
- name: macOS arm64 & Windows
+ name: ${{ matrix.os == 'macos-latest' && 'macOS arm64' || 'Windows x86_64' }} (${{ matrix.python-tag }})
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
- python-version: ["3.10"]
os: [macos-latest, windows-latest]
+ python-tag: ["abi3", "3.13t", "3.14t"]
steps:
- uses: actions/checkout@v6
@@ -305,7 +298,14 @@ jobs:
- name: Cache Cargo
uses: Swatinem/rust-cache@v2
with:
- key: ${{ inputs.build_mode }}
+ key: ${{ inputs.build_mode }}-${{ matrix.python-tag }}
+
+ - name: Setup Python (free-threaded)
+ if: matrix.python-tag != 'abi3'
+ uses: actions/setup-python@v6
+ with:
+ python-version: ${{ matrix.python-tag }}
+ freethreaded: true
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b
with:
@@ -318,22 +318,22 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install dependencies
+ if: matrix.python-tag == 'abi3'
run: uv sync --dev --no-install-package datafusion
- # Run clippy BEFORE maturin so we can avoid rebuilding. The features must match
- # exactly the features used by maturin. Linux maturin builds need to happen in a
- # container so only run this for our mac runner.
+ # Clippy is interpreter-agnostic; run once per OS (against the abi3 entry)
+ # so the matrix doesn't pay the cost three times.
- name: Run Clippy
- if: matrix.os != 'windows-latest'
+ if: matrix.os != 'windows-latest' && matrix.python-tag == 'abi3'
run: cargo clippy --no-deps --all-targets --features substrait -- -D warnings
- - name: Build Python package (release mode)
- if: inputs.build_mode == 'release'
- run: uv run --no-project maturin build --release --strip --features substrait
-
- - name: Build Python package (debug mode)
- if: inputs.build_mode != 'release'
- run: uv run --no-project maturin build --features substrait
+ - name: Build wheel
+ uses: ./.github/actions/build-wheel
+ with:
+ python-tag: ${{ matrix.python-tag }}
+ build-mode: ${{ inputs.build_mode }}
+ features: "substrait"
+ out-dir: "target/wheels"
- name: List Windows wheels
if: matrix.os == 'windows-latest'
@@ -350,7 +350,7 @@ jobs:
uses: actions/upload-artifact@v7
if: inputs.build_mode == 'release'
with:
- name: dist-${{ matrix.os }}
+ name: dist-${{ matrix.os }}-${{ matrix.python-tag }}
path: target/wheels/*
# ============================================
@@ -359,11 +359,12 @@ jobs:
build-macos-x86_64:
if: inputs.build_mode == 'release'
needs: [generate-license, lint-rust, lint-python]
+ name: macOS x86_64 (${{ matrix.python-tag }})
runs-on: macos-15-intel
strategy:
fail-fast: false
matrix:
- python-version: ["3.10"]
+ python-tag: ["abi3", "3.13t", "3.14t"]
steps:
- uses: actions/checkout@v6
@@ -379,7 +380,14 @@ jobs:
- name: Cache Cargo
uses: Swatinem/rust-cache@v2
with:
- key: ${{ inputs.build_mode }}
+ key: ${{ inputs.build_mode }}-${{ matrix.python-tag }}
+
+ - name: Setup Python (free-threaded)
+ if: matrix.python-tag != 'abi3'
+ uses: actions/setup-python@v6
+ with:
+ python-version: ${{ matrix.python-tag }}
+ freethreaded: true
- uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b
with:
@@ -392,11 +400,16 @@ jobs:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install dependencies
+ if: matrix.python-tag == 'abi3'
run: uv sync --dev --no-install-package datafusion
- - name: Build (release mode)
- run: |
- uv run --no-project maturin build --release --strip --features substrait
+ - name: Build wheel
+ uses: ./.github/actions/build-wheel
+ with:
+ python-tag: ${{ matrix.python-tag }}
+ build-mode: ${{ inputs.build_mode }}
+ features: "substrait"
+ out-dir: "target/wheels"
- name: List Mac wheels
run: find target/wheels/
@@ -404,7 +417,7 @@ jobs:
- name: Archive wheels
uses: actions/upload-artifact@v7
with:
- name: dist-macos-aarch64
+ name: dist-macos-aarch64-${{ matrix.python-tag }}
path: target/wheels/*
# ============================================
@@ -509,11 +522,12 @@ jobs:
with:
enable-cache: true
- # Download the Linux wheel built in the previous job
+ # Download the Linux wheel built in the previous job.
+ # Docs only need the abi3 wheel — interpreter doesn't matter for sphinx.
- name: Download pre-built Linux wheel
uses: actions/download-artifact@v8
with:
- name: dist-manylinux-x86_64
+ name: dist-manylinux-x86_64-abi3
path: wheels/
# Install from the pre-built wheels
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 2cd792ea..0c8fa4f7 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -15,8 +15,9 @@
# specific language governing permissions and limitations
# under the License.
-# Reusable workflow for running tests
-# This ensures the same tests run for both debug (PRs) and release (main/tags) builds
+# Reusable workflow for running tests.
+# Single matrix covers both GIL (abi3 wheel) and free-threaded
+# (per-interpreter wheels) builds.
name: Test
@@ -35,61 +36,69 @@ jobs:
strategy:
fail-fast: false
matrix:
- python-version:
- - "3.10"
- - "3.11"
- - "3.12"
- - "3.13"
- - "3.14"
- toolchain:
- - "stable"
-
+ include:
+ # GIL builds — all share the same abi3 wheel.
+ - { python-version: "3.10", wheel-tag: "abi3", freethreaded: false }
+ - { python-version: "3.11", wheel-tag: "abi3", freethreaded: false }
+ - { python-version: "3.12", wheel-tag: "abi3", freethreaded: false }
+ - { python-version: "3.13", wheel-tag: "abi3", freethreaded: false }
+ - { python-version: "3.14", wheel-tag: "abi3", freethreaded: false }
+ # Free-threaded builds — one wheel per interpreter.
+ - { python-version: "3.13t", wheel-tag: "3.13t", freethreaded: true }
+ - { python-version: "3.14t", wheel-tag: "3.14t", freethreaded: true }
steps:
- uses: actions/checkout@v6
- name: Setup Python
+ id: setup-python
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python-version }}
+ freethreaded: ${{ matrix.freethreaded }}
- name: Cache Cargo
uses: actions/cache@v5
with:
path: ~/.cargo
- key: cargo-cache-${{ matrix.toolchain }}-${{ hashFiles('Cargo.lock') }}
+ key: cargo-cache-stable-${{ hashFiles('Cargo.lock') }}
- name: Install dependencies
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b
with:
enable-cache: true
- # Download the Linux wheel built in the build workflow
- name: Download pre-built Linux wheel
uses: actions/download-artifact@v8
with:
- name: dist-manylinux-x86_64
+ name: dist-manylinux-x86_64-${{ matrix.wheel-tag }}
path: wheels/
- # Download the FFI test wheel
+ # FFI test wheel only built once (under the abi3 matrix entry in build.yml).
- name: Download pre-built FFI test wheel
+ if: matrix.wheel-tag == 'abi3'
uses: actions/download-artifact@v8
with:
name: test-ffi-manylinux-x86_64
path: wheels/
- # Install from the pre-built wheels
- name: Install from pre-built wheels
run: |
set -x
- uv venv
- # Install development dependencies
- uv sync --dev --no-install-package datafusion
- # Install all pre-built wheels
+ # Create the venv with the setup-python interpreter, then point
+ # every uv command explicitly at the venv's own interpreter.
+ # uv's interpreter discovery skips free-threaded builds unless
+ # asked by exact path, so plain `uv sync` (even with an activated
+ # venv or --active) re-picks the system 3.12 and recreates .venv.
+ # Targeting .venv/bin/python keeps sync and pip install in the
+ # same 3.13t/3.14t environment as the cp313t/cp314t wheel.
+ uv venv --python "${{ steps.setup-python.outputs.python-path }}"
+ VENV_PY="$PWD/.venv/bin/python"
+ uv sync --python "$VENV_PY" --dev --no-install-package datafusion
WHEELS=$(find wheels/ -name "*.whl")
if [ -n "$WHEELS" ]; then
echo "Installing wheels:"
echo "$WHEELS"
- uv pip install wheels/*.whl
+ uv pip install --python "$VENV_PY" wheels/*.whl
else
echo "ERROR: No wheels found!"
exit 1
@@ -98,16 +107,24 @@ jobs:
- name: Run tests
env:
RUST_BACKTRACE: 1
+ # On free-threaded interpreters, fail loud if any C extension
+ # re-enables the GIL implicitly.
+ PYTHON_GIL: ${{ matrix.freethreaded && '0' || '' }}
run: |
git submodule update --init
- uv run --no-project pytest -v --import-mode=importlib
+ # Use the .venv interpreter directly; uv discovery would skip the
+ # free-threaded build and re-pick the system 3.12 (see install step).
+ uv run --python "$PWD/.venv/bin/python" --no-project pytest -v --import-mode=importlib
+ # FFI + TPC-H examples only need to run once; gate to abi3 entries.
- name: FFI unit tests
+ if: matrix.wheel-tag == 'abi3'
run: |
cd examples/datafusion-ffi-example
uv run --no-project pytest python/tests/_test*.py
- name: Run tpchgen-cli to create 1 Gb dataset
+ if: matrix.wheel-tag == 'abi3'
run: |
mkdir examples/tpch/data
cd examples/tpch/data
@@ -115,6 +132,7 @@ jobs:
uv run --no-project tpchgen-cli -s 1 --format=parquet
- name: Run TPC-H examples
+ if: matrix.wheel-tag == 'abi3'
run: |
cd examples/tpch
uv run --no-project pytest _tests.py
diff --git a/Cargo.lock b/Cargo.lock
index 6a1ef244..d3cedb62 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2904,6 +2904,7 @@ version = "0.28.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e368e7ddfdeb98c9bca7f8383be1648fd84ab466bf2bc015e94008db6d35611e"
dependencies = [
+ "python3-dll-a",
"target-lexicon",
]
@@ -2953,6 +2954,15 @@ dependencies = [
"syn 2.0.117",
]
+[[package]]
+name = "python3-dll-a"
+version = "0.2.15"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d80ba7540edb18890d444c5aa8e1f1f99b1bdf26fb26ae383135325f4a36042b"
+dependencies = [
+ "cc",
+]
+
[[package]]
name = "quick-xml"
version = "0.39.4"
diff --git a/crates/core/Cargo.toml b/crates/core/Cargo.toml
index d714dc97..1f5b4e30 100644
--- a/crates/core/Cargo.toml
+++ b/crates/core/Cargo.toml
@@ -42,8 +42,7 @@ tokio = { workspace = true, features = [
] }
pyo3 = { workspace = true, features = [
"extension-module",
- "abi3",
- "abi3-py310",
+ "generate-import-lib",
] }
pyo3-async-runtimes = { workspace = true, features = ["tokio-runtime"] }
pyo3-log = { workspace = true }
@@ -74,7 +73,11 @@ prost-types = { workspace = true }
pyo3-build-config = { workspace = true }
[features]
-default = ["mimalloc"]
+default = ["mimalloc", "abi3"]
+# Stable ABI build — single wheel covers Python 3.10..3.14 (GIL builds only).
+# Mutually exclusive with free-threaded interpreters (cp313t / cp314t); the
+# free-threaded wheel build must pass --no-default-features.
+abi3 = ["pyo3/abi3", "pyo3/abi3-py310"]
protoc = ["datafusion-substrait/protoc"]
substrait = ["dep:datafusion-substrait"]
diff --git a/dev/release/README.md b/dev/release/README.md
index 1b02ca57..384ef921 100644
--- a/dev/release/README.md
+++ b/dev/release/README.md
@@ -75,6 +75,7 @@ We maintain a `CHANGELOG.md` so our users know what has been changed between rel
The changelog is generated using a Python script:
```bash
+$ uv sync --group release
$ GITHUB_TOKEN=<TOKEN> ./dev/release/generate-changelog.py 52.0.0 HEAD 53.0.0 > dev/changelog/53.0.0.md
```
diff --git a/pyproject.toml b/pyproject.toml
index 418640a4..2b6a976d 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -40,6 +40,7 @@ classifiers = [
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
+ "Programming Language :: Python :: Free Threading :: 2 - Beta",
"Programming Language :: Python",
"Programming Language :: Rust",
]
@@ -201,7 +202,6 @@ dev = [
"numpy>=2.3.2;python_version>='3.14'",
"pre-commit>=4.3.0",
"pyarrow>=19.0.0",
- "pygithub==2.5.0",
"pytest-asyncio>=0.23.3",
"pytest-timeout>=2.3.1",
"pytest>=7.4.4",
@@ -209,6 +209,10 @@ dev = [
"ruff>=0.15.1",
"toml>=0.10.2",
]
+# Release tooling only. Kept out of `dev` because pygithub pulls in
+# cryptography, which ships no free-threaded wheel and fails to build
+# from sdist under free-threaded interpreters (PyO3 < 3.14 support).
+release = ["pygithub==2.5.0"]
docs = [
"ipython>=8.12.3",
"jinja2>=3.1.5",
diff --git a/uv.lock b/uv.lock
index 26ab8b20..6673b7fe 100644
--- a/uv.lock
+++ b/uv.lock
@@ -340,7 +340,6 @@ dev = [
{ name = "numpy", version = "2.3.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.14'" },
{ name = "pre-commit" },
{ name = "pyarrow" },
- { name = "pygithub" },
{ name = "pytest" },
{ name = "pytest-asyncio" },
{ name = "pytest-timeout" },
@@ -359,6 +358,9 @@ docs = [
{ name = "sphinx" },
{ name = "sphinx-autoapi" },
]
+release = [
+ { name = "pygithub" },
+]
[package.metadata]
requires-dist = [
@@ -378,7 +380,6 @@ dev = [
{ name = "numpy", marker = "python_full_version >= '3.14'", specifier = ">=2.3.2" },
{ name = "pre-commit", specifier = ">=4.3.0" },
{ name = "pyarrow", specifier = ">=19.0.0" },
- { name = "pygithub", specifier = "==2.5.0" },
{ name = "pytest", specifier = ">=7.4.4" },
{ name = "pytest-asyncio", specifier = ">=0.23.3" },
{ name = "pytest-timeout", specifier = ">=2.3.1" },
@@ -397,6 +398,7 @@ docs = [
{ name = "sphinx", specifier = ">=7.1.2" },
{ name = "sphinx-autoapi", specifier = ">=3.4.0" },
]
+release = [{ name = "pygithub", specifier = "==2.5.0" }]
[[package]]
name = "decorator"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org