You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/10/14 02:33:38 UTC
(spark) branch master updated: [SPARK-60210][INFRA] Add scheduled Python 3.15 build
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3d4eb2e2a525 [SPARK-60210][INFRA] Add scheduled Python 3.15 build
3d4eb2e2a525 is described below
commit 3d4eb2e2a525add06cda9f59d8b6b99ed4e759b9
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Wed Oct 14 10:33:16 2026 +0800
[SPARK-60210][INFRA] Add scheduled Python 3.15 build
### What changes were proposed in this pull request?
Add a scheduled Python 3.15 PySpark build on master, with a matching CI image and image cache build. The workflow runs every two days at 22:00 UTC and supports manual dispatch. Add its status badge to the README.
The Python 3.15 image initially installs a small set of dependencies and the scheduled job runs non-pandas PySpark tests. The broader dependency groups currently select pandas 2.3.3, which has no Python 3.15 wheel and fails to build from source in the image.
JIRA: [SPARK-60210](https://issues.apache.org/jira/browse/SPARK-60210)
### Why are the changes needed?
A scheduled build provides regular compatibility feedback for PySpark on Python 3.15. The new Python 3.15 CI is currently failing, but we need to keep it enabled to monitor compatibility and track progress toward support.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Checked the changes with `git diff --check` and scanned the new files for non-ASCII characters and long lines. The first PR image build failed while compiling pandas 2.3.3. The follow-up Python 3.15 image build passed with the reduced dependency set; the new Python 3.15 CI is currently failing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59421 from zhengruifeng/py-python315-scheduled-job-dev-7.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
.github/workflows/build_infra_images_cache.yml | 18 +++++++
.github/workflows/build_python_3.15.yml | 46 ++++++++++++++++
README.md | 1 +
dev/spark-test-image/python-315/Dockerfile | 73 ++++++++++++++++++++++++++
4 files changed, 138 insertions(+)
diff --git a/.github/workflows/build_infra_images_cache.yml b/.github/workflows/build_infra_images_cache.yml
index 6e3000570617..805fc6efb9f8 100644
--- a/.github/workflows/build_infra_images_cache.yml
+++ b/.github/workflows/build_infra_images_cache.yml
@@ -43,6 +43,7 @@ on:
- 'dev/spark-test-image/python-312-pandas-3/Dockerfile'
- 'dev/spark-test-image/python-313/Dockerfile'
- 'dev/spark-test-image/python-314/Dockerfile'
+ - 'dev/spark-test-image/python-315/Dockerfile'
- 'dev/spark-test-image/python-314-nogil/Dockerfile'
- '.github/workflows/build_infra_images_cache.yml'
- 'pyproject.toml'
@@ -226,6 +227,22 @@ jobs:
- name: Image digest (PySpark with Python 3.14)
if: hashFiles('dev/spark-test-image/python-314/Dockerfile') != ''
run: echo ${{ steps.docker_build_pyspark_python_314.outputs.digest }}
+ - name: Build and push (PySpark with Python 3.15)
+ if: hashFiles('dev/spark-test-image/python-315/Dockerfile') != ''
+ id: docker_build_pyspark_python_315
+ continue-on-error: true
+ uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a
+ with:
+ context: ./dev/spark-test-image/python-315/
+ build-contexts: |
+ root=./
+ push: true
+ tags: ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-315-cache:${{ github.ref_name }}-static
+ cache-from: type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-315-cache:${{ github.ref_name }}
+ cache-to: type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-315-cache:${{ github.ref_name }},mode=max
+ - name: Image digest (PySpark with Python 3.15)
+ if: hashFiles('dev/spark-test-image/python-315/Dockerfile') != ''
+ run: echo ${{ steps.docker_build_pyspark_python_315.outputs.digest }}
- name: Build and push (PySpark with Python 3.14 no GIL)
if: hashFiles('dev/spark-test-image/python-314-nogil/Dockerfile') != ''
id: docker_build_pyspark_python_314_nogil
@@ -264,6 +281,7 @@ jobs:
check "pyspark-python-312-pandas-3" "${{ steps.docker_build_pyspark_python_312_pandas_3.outcome }}"
check "pyspark-python-313" "${{ steps.docker_build_pyspark_python_313.outcome }}"
check "pyspark-python-314" "${{ steps.docker_build_pyspark_python_314.outcome }}"
+ check "pyspark-python-315" "${{ steps.docker_build_pyspark_python_315.outcome }}"
check "pyspark-python-314-nogil" "${{ steps.docker_build_pyspark_python_314_nogil.outcome }}"
if [ "$status" -ne 0 ]; then
echo "::error::One or more image builds failed; see entries above marked 'failure'."
diff --git a/.github/workflows/build_python_3.15.yml b/.github/workflows/build_python_3.15.yml
new file mode 100644
index 000000000000..2c8535553f28
--- /dev/null
+++ b/.github/workflows/build_python_3.15.yml
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: "Build / Python-only (Python 3.15)"
+
+on:
+ schedule:
+ - cron: '0 22 */2 * *'
+ workflow_dispatch:
+
+jobs:
+ run-build:
+ permissions:
+ packages: write
+ name: Run
+ uses: ./.github/workflows/build_and_test.yml
+ if: github.repository == 'apache/spark'
+ with:
+ java: 17
+ branch: master
+ hadoop: hadoop3
+ envs: >-
+ {
+ "PYSPARK_IMAGE_TO_TEST": "python-315",
+ "PYTHON_TO_TEST": "python3.15"
+ }
+ jobs: >-
+ {
+ "pyspark": "true"
+ }
diff --git a/README.md b/README.md
index b57b53688f54..133112688abe 100644
--- a/README.md
+++ b/README.md
@@ -51,6 +51,7 @@ This README file only contains basic setup instructions.
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.13.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14_nogil.yml) |
+| | [](https://github.com/apache/spark/actions/workflows/build_python_3.15.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_minimum.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_connect40.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_connect.yml) |
diff --git a/dev/spark-test-image/python-315/Dockerfile b/dev/spark-test-image/python-315/Dockerfile
new file mode 100644
index 000000000000..7d8faaa68160
--- /dev/null
+++ b/dev/spark-test-image/python-315/Dockerfile
@@ -0,0 +1,73 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Image for building and testing Spark branches. Based on Ubuntu 24.04.
+# See also in https://hub.docker.com/_/ubuntu
+FROM ubuntu:noble
+LABEL org.opencontainers.image.authors="Apache Spark project <de...@spark.apache.org>"
+LABEL org.opencontainers.image.licenses="Apache-2.0"
+LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark with Python 3.15"
+# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
+LABEL org.opencontainers.image.version=""
+
+ENV FULL_REFRESH_DATE=20261013
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV DEBCONF_NONINTERACTIVE_SEEN=true
+
+RUN printf 'Types: deb\nURIs: https://mirrors.edge.kernel.org/ubuntu\nSuites: noble noble-updates noble-security\nComponents: main restricted universe multiverse\nSigned-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg\n' > /etc/apt/sources.list.d/mirror.sources
+
+RUN apt-get update && apt-get install -y \
+ build-essential \
+ ca-certificates \
+ curl \
+ gfortran \
+ git \
+ gnupg \
+ libgit2-dev \
+ liblapack-dev \
+ libopenblas-dev \
+ libssl-dev \
+ openjdk-17-jdk-headless \
+ pkg-config \
+ tzdata \
+ software-properties-common \
+ zlib1g-dev \
+ zstd
+
+# Install Python 3.15
+RUN add-apt-repository ppa:deadsnakes/ppa
+RUN apt-get update && apt-get install -y \
+ python3.15 \
+ python3.15-venv \
+ && apt-get autoremove --purge -y \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+# Setup virtual environment
+ENV VIRTUAL_ENV=/opt/spark-venv
+RUN python3.15 -m venv $VIRTUAL_ENV
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
+COPY --from=root pyproject.toml ./pyproject.toml
+
+RUN python3.15 -m pip install -U pip
+
+# pandas<3 does not have Python 3.15 wheels yet. Start with dependencies that can be installed.
+RUN python3.15 -m pip install --group ml_torch --index-url https://download.pytorch.org/whl/cpu && \
+ python3.15 -m pip install py4j psutil six unittest-xml-reporting && \
+ python3.15 -m pip cache purge
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org