You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/06/08 09:15:53 UTC
(spark) branch branch-4.2 updated: [SPARK-57278][INFRA] Install zstd in CI container images to fix GitHub Actions cache
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 06fad3633d26 [SPARK-57278][INFRA] Install zstd in CI container images to fix GitHub Actions cache
06fad3633d26 is described below
commit 06fad3633d263b684ef271cbbf196ed78524b33a
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Jun 8 17:15:14 2026 +0800
[SPARK-57278][INFRA] Install zstd in CI container images to fix GitHub Actions cache
### What changes were proposed in this pull request?
Install `zstd` in all CI container image Dockerfiles (`dev/infra/Dockerfile` and the `python-*`, `docs`, `lint`, `sparkr` images under `dev/spark-test-image/`).
### Why are the changes needed?
`actions/cache` has never successfully restored a cache in any container-based CI job — confirmed by `apache/spark`'s cache history, which has no `pyspark-coursier-*` / `sparkr-coursier-*` / `docs-coursier-*` entry. This is a long-standing issue, present since container jobs were introduced.
`actions/cache` computes a cache **version** = `SHA256(paths + compression_method)` and includes it in the lookup URL. Host runners have `zstd` and use it; container images lack `zstd` and fall back to `gzip`. The version then differs, so caches saved by host jobs (e.g. the Coursier cache written by `precompile`) are invisible to container jobs even when the key matches. Installing `zstd` aligns the compression method.
(The `build-` cache happened to work because it is written by both host and container jobs, so a gzip-version entry also existed; host-only caches had no gzip entry.)
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
Before (run [26956300346](https://github.com/zhengruifeng/spark/actions/runs/26956300346)): `precompile` saved `Linux-coursier-<hash>`, but all container jobs reported `Cache not found` for the same key minutes later.
After (run [26996424034](https://github.com/zhengruifeng/spark/actions/runs/26996424034)): `pyspark`, `sparkr`, `lint`, `docs` all `Cache restored from key: Linux-coursier-<hash>`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-sonnet-4-6)
Closes #56324 from zhengruifeng/fix-container-coursier-cache-ci-cache-opt-dev6.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
(cherry picked from commit 9c1adaf8aba2fb507e32a318f4018e58d56861df)
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
dev/infra/Dockerfile | 1 +
dev/spark-test-image/docs/Dockerfile | 1 +
dev/spark-test-image/lint/Dockerfile | 1 +
dev/spark-test-image/python-311/Dockerfile | 3 ++-
dev/spark-test-image/python-312-classic-only/Dockerfile | 1 +
dev/spark-test-image/python-312-pandas-3/Dockerfile | 1 +
dev/spark-test-image/python-312/Dockerfile | 1 +
dev/spark-test-image/python-313/Dockerfile | 3 ++-
dev/spark-test-image/python-314-nogil/Dockerfile | 3 ++-
dev/spark-test-image/python-314/Dockerfile | 3 ++-
dev/spark-test-image/python-minimum/Dockerfile | 3 ++-
dev/spark-test-image/sparkr/Dockerfile | 1 +
12 files changed, 17 insertions(+), 5 deletions(-)
diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile
index b848f8eb754d..57cde202dde8 100644
--- a/dev/infra/Dockerfile
+++ b/dev/infra/Dockerfile
@@ -70,6 +70,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/docs/Dockerfile b/dev/spark-test-image/docs/Dockerfile
index 3b02e2ae1cff..f58168627dd1 100644
--- a/dev/spark-test-image/docs/Dockerfile
+++ b/dev/spark-test-image/docs/Dockerfile
@@ -70,6 +70,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& apt-get autoremove --purge -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/lint/Dockerfile b/dev/spark-test-image/lint/Dockerfile
index c6bf389ecb22..b7cecae654cb 100644
--- a/dev/spark-test-image/lint/Dockerfile
+++ b/dev/spark-test-image/lint/Dockerfile
@@ -63,6 +63,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& apt-get autoremove --purge -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/python-311/Dockerfile b/dev/spark-test-image/python-311/Dockerfile
index 8e5044aeb954..e39e9fac70d2 100644
--- a/dev/spark-test-image/python-311/Dockerfile
+++ b/dev/spark-test-image/python-311/Dockerfile
@@ -46,7 +46,8 @@ RUN apt-get update && apt-get install -y \
pkg-config \
tzdata \
software-properties-common \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.11
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/python-312-classic-only/Dockerfile b/dev/spark-test-image/python-312-classic-only/Dockerfile
index da2b99cd7838..ceb4694b2dc9 100644
--- a/dev/spark-test-image/python-312-classic-only/Dockerfile
+++ b/dev/spark-test-image/python-312-classic-only/Dockerfile
@@ -49,6 +49,7 @@ RUN apt-get update && apt-get install -y \
tzdata \
software-properties-common \
zlib1g-dev \
+ zstd \
&& apt-get autoremove --purge -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/python-312-pandas-3/Dockerfile b/dev/spark-test-image/python-312-pandas-3/Dockerfile
index c54a8c284a6b..e2a2c189df15 100644
--- a/dev/spark-test-image/python-312-pandas-3/Dockerfile
+++ b/dev/spark-test-image/python-312-pandas-3/Dockerfile
@@ -52,6 +52,7 @@ RUN apt-get update && apt-get install -y \
tzdata \
software-properties-common \
zlib1g-dev \
+ zstd \
&& apt-get autoremove --purge -y \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
diff --git a/dev/spark-test-image/python-312/Dockerfile b/dev/spark-test-image/python-312/Dockerfile
index 8eee9e414f1d..72b40510b8db 100644
--- a/dev/spark-test-image/python-312/Dockerfile
+++ b/dev/spark-test-image/python-312/Dockerfile
@@ -48,6 +48,7 @@ RUN apt-get update && apt-get install -y \
pkg-config \
tzdata \
software-properties-common \
+ zstd \
zlib1g-dev \
&& apt-get autoremove --purge -y \
&& apt-get clean \
diff --git a/dev/spark-test-image/python-313/Dockerfile b/dev/spark-test-image/python-313/Dockerfile
index 6cfdd2d5a86e..c13e364f1589 100644
--- a/dev/spark-test-image/python-313/Dockerfile
+++ b/dev/spark-test-image/python-313/Dockerfile
@@ -46,7 +46,8 @@ RUN apt-get update && apt-get install -y \
pkg-config \
tzdata \
software-properties-common \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.13
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/python-314-nogil/Dockerfile b/dev/spark-test-image/python-314-nogil/Dockerfile
index edfea3172992..6dea9c2fc35e 100644
--- a/dev/spark-test-image/python-314-nogil/Dockerfile
+++ b/dev/spark-test-image/python-314-nogil/Dockerfile
@@ -46,7 +46,8 @@ RUN apt-get update && apt-get install -y \
pkg-config \
tzdata \
software-properties-common \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.14 (no GIL)
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/python-314/Dockerfile b/dev/spark-test-image/python-314/Dockerfile
index b6bc76c6b93b..2f3570ec0687 100644
--- a/dev/spark-test-image/python-314/Dockerfile
+++ b/dev/spark-test-image/python-314/Dockerfile
@@ -46,7 +46,8 @@ RUN apt-get update && apt-get install -y \
pkg-config \
tzdata \
software-properties-common \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.14
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/python-minimum/Dockerfile b/dev/spark-test-image/python-minimum/Dockerfile
index 72abd5f5811b..d2e4a83ce81e 100644
--- a/dev/spark-test-image/python-minimum/Dockerfile
+++ b/dev/spark-test-image/python-minimum/Dockerfile
@@ -47,7 +47,8 @@ RUN apt-get update && apt-get install -y \
pkg-config \
tzdata \
software-properties-common \
- zlib1g-dev
+ zlib1g-dev \
+ zstd
# Install Python 3.10
RUN add-apt-repository ppa:deadsnakes/ppa
diff --git a/dev/spark-test-image/sparkr/Dockerfile b/dev/spark-test-image/sparkr/Dockerfile
index 07816add74fc..8a03095aef7e 100644
--- a/dev/spark-test-image/sparkr/Dockerfile
+++ b/dev/spark-test-image/sparkr/Dockerfile
@@ -62,6 +62,7 @@ RUN apt-get update && apt-get install -y \
software-properties-common \
wget \
zlib1g-dev \
+ zstd \
&& rm -rf /var/lib/apt/lists/*
RUN echo 'deb https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/' >> /etc/apt/sources.list
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org