You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@livy.apache.org by gg...@apache.org on 2026/08/20 15:06:13 UTC
(livy) branch master updated: [LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version
This is an automated email from the ASF dual-hosted git repository.
gyogal pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/livy.git
The following commit(s) were added to refs/heads/master by this push:
new 3d2a94d7 [LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version
3d2a94d7 is described below
commit 3d2a94d7507dc8dab0bd8c8b2da42a9903764346
Author: Gabor Roczei <19...@users.noreply.github.com>
AuthorDate: Thu Aug 20 17:06:07 2026 +0200
[LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version
## What changes were proposed in this pull request?
Prepares the CI environment and the python-api package metadata for the
upcoming Spark 4 support commit. Both changes are also useful on the
existing Spark 3 build: the PEP 440 fix unblocks `pip3 install
livy-python-api` on modern pip (>= 24), and the Python bump keeps the
CI image on a version supported by both Spark 3.5 and Spark 4.1.
**CI Python bumped to 3.11.11**
dev/docker/livy-dev-base/Dockerfile pyenv Python bumped from 3.9.21 to
3.11.11, and .github/workflows/integration-tests.yaml pins
`pyenv global 3.11.11` explicitly. Rationale (per each Spark release's
python/setup.py): Spark 4.1 declares `python_requires=">=3.10"` with
classifiers listing 3.10/3.11/3.12/3.13/3.14, while Spark 3.5 declares
`python_requires=">=3.8"` with classifiers listing 3.8/3.9/3.10/3.11.
The intersection of the two supported ranges is 3.10 and 3.11; we pick
the higher one (3.11.11) so the same image serves both matrix profiles.
The old 3.9.21 was below the Spark 4.1 lower bound. Both the Dockerfile
and the workflow now document this choice inline so future bumps stay
in sync.
The CI Docker image (`ghcr.io/${owner}/livy-ci:latest`) had to be
rebuilt and pushed manually for the PR CI to pick up the new Python
before this PR is merged: `.github/workflows/build-ci-image.yaml` only
fires on push to master, so branch/PR runs would otherwise still pull
the stale image cached from the previous Dockerfile. Commands used
(from macOS Apple Silicon, cross-built for linux/amd64 to match the
GitHub Actions runners):
gh auth refresh --scopes write:packages,read:packages
gh auth token | docker login ghcr.io -u <owner> --password-stdin
docker buildx build --platform linux/amd64 \
-t ghcr.io/<owner>/livy-ci:latest --push \
dev/docker/livy-dev-base
After this PR merges to master, the workflow will republish the image
on any future Dockerfile change automatically, so this manual step is
only needed for the bootstrap run.
**python-api/setup.py PEP 440 version fix**
Version bumped from `1.0.0-SNAPSHOT` (Maven-style, not PEP 440
compliant) to `1.0.0.dev0` (the canonical Python "pre-release under
active development" form). Modern pip (>= 24) refuses to parse the old
value with `Invalid version: '1.0.0-SNAPSHOT'`, which surfaced as
`WARNING: Error parsing dependencies of livy-python-api` on every
`pip3 install` and blocked the `validate Python-API requests`
integration test from resolving its dependencies. The Maven POM
version (`python-api/pom.xml`) stays `1.0.0-SNAPSHOT` -- Maven and pip
have separate versioning conventions and only the pip-visible metadata
needs to change.
## How was this patch tested?
- `pip3 install ./python-api` no longer emits
`WARNING: Error parsing dependencies of livy-python-api: Invalid
version: '1.0.0-SNAPSHOT'`; the package installs cleanly on pip 24+.
- Rebuilt the CI Docker image locally with the pyenv 3.11.11 bump and
confirmed `python3 --version` reports `Python 3.11.11` inside the
container.
- GitHub Actions Integration Tests workflow runs `pyenv global 3.11.11`
successfully on the -Pspark3 -Pscala-2.12 matrix entry (this commit
does not yet add any Spark 4 entry).
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)
---
.github/workflows/integration-tests.yaml | 7 ++++++-
dev/docker/livy-dev-base/Dockerfile | 13 +++++++++++--
python-api/setup.py | 6 +++++-
3 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/integration-tests.yaml b/.github/workflows/integration-tests.yaml
index 968937e0..c1dd01b9 100644
--- a/.github/workflows/integration-tests.yaml
+++ b/.github/workflows/integration-tests.yaml
@@ -48,8 +48,13 @@ jobs:
restore-keys: |
${{ runner.os }}-maven-
-
+ # Must match the pyenv-installed version in dev/docker/livy-dev-base/Dockerfile.
+ # Per each Spark release's python/setup.py: Spark 4.1 supports 3.10..3.14
+ # (python_requires=">=3.10"), Spark 3.5 supports 3.8..3.11
+ # (python_requires=">=3.8"). The intersection is 3.10 and 3.11; 3.11.11
+ # is picked as the shared version.
name: Set Python 3 as default
- run: pyenv global 3 && echo "PYSPARK_PYTHON=$(which python3)" >> "$GITHUB_ENV"
+ run: pyenv global 3.11.11 && echo "PYSPARK_PYTHON=$(which python3)" >> "$GITHUB_ENV"
-
name: Set JDK version
run: update-alternatives --set java ${{ matrix.jdk_path }}
diff --git a/dev/docker/livy-dev-base/Dockerfile b/dev/docker/livy-dev-base/Dockerfile
index 5686e61d..b0d06b73 100644
--- a/dev/docker/livy-dev-base/Dockerfile
+++ b/dev/docker/livy-dev-base/Dockerfile
@@ -70,8 +70,17 @@ RUN git clone https://github.com/pyenv/pyenv.git $HOME/pyenv
ENV PYENV_ROOT=$HOME/pyenv
ENV PATH="$HOME/pyenv/shims:$HOME/pyenv/bin:$HOME/bin:$PATH"
-RUN pyenv install -v 3.9.21 && \
- pyenv global 3.9.21 && \
+# Python 3.11.11 is chosen because it is in the officially supported range of
+# BOTH matrix profiles, per each Spark release's python/setup.py:
+# * -Pspark4 (Spark 4.1.2): python_requires=">=3.10", classifiers list
+# 3.10 / 3.11 / 3.12 / 3.13 / 3.14.
+# * -Pspark3 (Spark 3.5.6): python_requires=">=3.8", classifiers list
+# 3.8 / 3.9 / 3.10 / 3.11.
+# The intersection is 3.10 and 3.11; we pick the higher one (3.11.11) so the
+# same image serves both profiles. When bumping this, keep
+# .github/workflows/integration-tests.yaml's `pyenv global` in sync.
+RUN pyenv install -v 3.11.11 && \
+ pyenv global 3.11.11 && \
pyenv rehash
# Install build dependencies for python3
diff --git a/python-api/setup.py b/python-api/setup.py
index 4d7b9b5f..df1a4de1 100644
--- a/python-api/setup.py
+++ b/python-api/setup.py
@@ -37,7 +37,11 @@ requirements = [
setup(
name='livy-python-api',
- version="1.0.0-SNAPSHOT",
+ # PEP 440 disallows Maven-style `-SNAPSHOT` suffixes; use `.dev0` instead,
+ # which is the canonical Python equivalent for "pre-release under active
+ # development". Modern pip (>= 24) rejects `1.0.0-SNAPSHOT` outright with
+ # `Invalid version: '1.0.0-SNAPSHOT'` when parsing this package.
+ version="1.0.0.dev0",
packages=["livy", "livy-tests"],
package_dir={
"": "src/main/python",