You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/04/22 22:39:31 UTC
(spark) branch branch-4.1 updated: [SPARK-56583][K8S] Skip `pip` upgrade and use `--break-system-packages` to avoid PEP 668 error in PySpark Dockerfile
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 07cbf4b6079c [SPARK-56583][K8S] Skip `pip` upgrade and use `--break-system-packages` to avoid PEP 668 error in PySpark Dockerfile
07cbf4b6079c is described below
commit 07cbf4b6079c0dd598f00f56b709402e89ac8122
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Apr 22 14:22:16 2026 -0700
[SPARK-56583][K8S] Skip `pip` upgrade and use `--break-system-packages` to avoid PEP 668 error in PySpark Dockerfile
This PR aims to (1) skip additional `pip` upgrade and (2) add the `--break-system-packages` flag to the `pip3 install` command in the PySpark Kubernetes Dockerfile (`resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile`).
Currently, Apache Spark CI is broken due to the K8s PySpark docker image building.
- **master**: https://github.com/apache/spark/actions/runs/24799280675/job/72577453527
- **branch-4.1**: https://github.com/apache/spark/actions/runs/24777943486/job/72501169416
With Python 3.12+ on Debian/Ubuntu-based images (e.g., Ubuntu Noble), the system Python enforces [PEP 668](https://peps.python.org/pep-0668/) and marks the environment as externally managed. As a result, `pip3 install --no-cache-dir --upgrade pip setuptools` fails during the Docker image build with:
```
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
...
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
```
Since the installation happens inside a dedicated container image where the system Python is only used for PySpark, it is safe to bypass the PEP 668 restriction with `--break-system-packages`.
No. This only restores the PySpark Kubernetes image build on Python 3.12+ base images.
Pass the CIs with the K8s integration test.
For the record, I also manually verified this patch with Apache Spark 4.2.0-preview4 like the following
```
$ bin/docker-image-tool.sh -r docker.io/apache -t 4.2.0-preview4 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
```
Generated-by: Claude Opus 4.7
Closes #55492 from dongjoon-hyun/SPARK-56583.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 2133f3242eb402cd708c219ac8ed1b558c045908)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../docker/src/main/dockerfiles/spark/bindings/python/Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
index 14ee735d0333..640247dd7408 100644
--- a/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
+++ b/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile
@@ -31,7 +31,7 @@ USER 0
RUN mkdir ${SPARK_HOME}/python
RUN apt-get update && \
apt install -y python3 python3-pip && \
- pip3 install --upgrade pip setuptools && \
+ pip3 install --break-system-packages --upgrade setuptools && \
# Removed the .cache to save space
rm -rf /root/.cache && rm -rf /var/cache/apt/* && rm -rf /var/lib/apt/lists/*
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org