You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/06/09 01:40:27 UTC
(spark) branch master updated: [SPARK-57330][INFRA] Switch shared CI compile artifacts to zstd compression
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ac0c117cf280 [SPARK-57330][INFRA] Switch shared CI compile artifacts to zstd compression
ac0c117cf280 is described below
commit ac0c117cf280e1a44e8fdc0c00a91740bb9a6abf
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Tue Jun 9 09:40:15 2026 +0800
[SPARK-57330][INFRA] Switch shared CI compile artifacts to zstd compression
### What changes were proposed in this pull request?
Compress the shared compile artifacts that CI passes between jobs with `zstd` instead of `gzip`, across the three reusable workflows that produce/consume them:
- `build_and_test.yml` - `compile-artifact` (produced by `precompile`, consumed by the `build`, `pyspark`, `sparkr`, `tpcds-1g`, `docker-integration-tests` and `k8s-integration-tests` jobs)
- `python_hosted_runner_test.yml` - `compile-artifact` (macOS / ARM python matrix)
- `maven_test.yml` - `compile-target` and `compile-m2-spark`
Each producer pipes `tar` through the `zstd` binary, and each consumer decompresses with `zstd -dc`:
```bash
# create
... | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
# extract
zstd -dc compile-artifact.tar.zst | tar -xf -
```
`zstd` is driven through the standalone binary rather than `tar --zstd` on purpose: GitHub's macOS runners ship bsdtar, whose `--zstd` hangs, so letting `tar` do plain (un)archiving and piping through the `zstd` binary keeps one portable idiom across Linux hosts, the container images, `ubuntu-24.04-arm` and macOS.
### Why are the changes needed?
`zstd` compresses and (especially) decompresses much faster than `gzip` at a comparable or better ratio, and `-T0` parallelizes compression across all cores. These artifacts are produced once and downloaded by up to 8 downstream jobs per run, so faster (de)compression shortens the critical path of every matrix entry. The `zstd` binary is already present on all relevant runners and container images (the latter since SPARK-57278), so no new dependency is introduced.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
- Local validation: all three workflows parse as YAML, and no `*.tar.gz` reference to these artifacts remains.
- The artifacts are ephemeral per run (`retention-days: 1`, names keyed by `run_id`), so producer and consumer always run the same code; there is no cross-version compatibility concern.
- The `build_and_test.yml` path is exercised by this PR's CI. The `maven_test.yml` and `python_hosted_runner_test.yml` paths run via their scheduled / `workflow_dispatch` callers and can be validated by dispatching those workflows on the fork.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #56369 from zhengruifeng/shared-artifact-zstd-dev6.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
.github/workflows/build_and_test.yml | 30 ++++++++++++-------------
.github/workflows/maven_test.yml | 20 ++++++++---------
.github/workflows/python_hosted_runner_test.yml | 10 ++++-----
3 files changed, 30 insertions(+), 30 deletions(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index 42325a6401e0..dcd91ac093dd 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -425,8 +425,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
@@ -636,13 +636,13 @@ jobs:
- name: Package compile output
run: |
find . -type d -name target -not -path './build/*' -not -path './.git/*' -print0 \
- | tar --null -czf compile-artifact.tar.gz -T -
- ls -lh compile-artifact.tar.gz
+ | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
+ ls -lh compile-artifact.tar.zst
- name: Upload compile artifact
uses: actions/upload-artifact@v7
with:
name: spark-compile-${{ inputs.branch }}-${{ github.run_id }}
- path: compile-artifact.tar.gz
+ path: compile-artifact.tar.zst
retention-days: 1
if-no-files-found: error
@@ -783,8 +783,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
@@ -956,8 +956,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
@@ -1499,8 +1499,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Cache TPC-DS generated data
id: cache-tpcds-sf-1
uses: actions/cache@v5
@@ -1631,8 +1631,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
run: |
@@ -1722,8 +1722,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
- name: Install R
run: |
sudo apt update
diff --git a/.github/workflows/maven_test.yml b/.github/workflows/maven_test.yml
index cc88fab6721f..a302da14ebc1 100644
--- a/.github/workflows/maven_test.yml
+++ b/.github/workflows/maven_test.yml
@@ -122,18 +122,18 @@ jobs:
# and the connect entry rebuilds it via `mvn install -pl assembly`.
find . \( -path './build' -o -path './.git' -o -path './assembly' \) -prune \
-o -type d -name target -print0 \
- | tar --null -czf compile-target.tar.gz -T -
+ | tar --null -cf - -T - | zstd -c -T0 > compile-target.tar.zst
if [ -d "$HOME/.m2/repository/org/apache/spark" ]; then
- tar -C "$HOME/.m2/repository/org/apache" -czf compile-m2-spark.tar.gz spark
+ tar -C "$HOME/.m2/repository/org/apache" -cf - spark | zstd -c -T0 > compile-m2-spark.tar.zst
fi
- ls -lh compile-target.tar.gz compile-m2-spark.tar.gz
+ ls -lh compile-target.tar.zst compile-m2-spark.tar.zst
- name: Upload compile artifact
uses: actions/upload-artifact@v7
with:
name: spark-maven-compile-${{ inputs.branch }}-java${{ inputs.java }}-${{ github.run_id }}
path: |
- compile-target.tar.gz
- compile-m2-spark.tar.gz
+ compile-target.tar.zst
+ compile-m2-spark.tar.zst
retention-days: 1
if-no-files-found: error
@@ -283,12 +283,12 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-target.tar.gz
- rm compile-target.tar.gz
- if [ -f compile-m2-spark.tar.gz ]; then
+ zstd -dc compile-target.tar.zst | tar -xf -
+ rm compile-target.tar.zst
+ if [ -f compile-m2-spark.tar.zst ]; then
mkdir -p "$HOME/.m2/repository/org/apache"
- tar -C "$HOME/.m2/repository/org/apache" -xzf compile-m2-spark.tar.gz
- rm compile-m2-spark.tar.gz
+ zstd -dc compile-m2-spark.tar.zst | tar -C "$HOME/.m2/repository/org/apache" -xf -
+ rm compile-m2-spark.tar.zst
fi
# Run the tests using script command.
# BSD's script command doesn't support -c option, and the usage is different from Linux's one.
diff --git a/.github/workflows/python_hosted_runner_test.yml b/.github/workflows/python_hosted_runner_test.yml
index 160fbd9bd551..a65f77cee22d 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -119,13 +119,13 @@ jobs:
- name: Package compile output
run: |
find . -type d -name target -not -path './build/*' -not -path './.git/*' -print0 \
- | tar --null -czf compile-artifact.tar.gz -T -
- ls -lh compile-artifact.tar.gz
+ | tar --null -cf - -T - | zstd -c -T0 > compile-artifact.tar.zst
+ ls -lh compile-artifact.tar.zst
- name: Upload compile artifact
uses: actions/upload-artifact@v6
with:
name: spark-compile-${{ inputs.os }}-${{ inputs.branch }}-${{ github.run_id }}
- path: compile-artifact.tar.gz
+ path: compile-artifact.tar.zst
retention-days: 1
if-no-files-found: error
@@ -246,8 +246,8 @@ jobs:
if: steps.download-precompiled.outcome == 'success'
continue-on-error: true
run: |
- tar -xzf compile-artifact.tar.gz
- rm compile-artifact.tar.gz
+ zstd -dc compile-artifact.tar.zst | tar -xf -
+ rm compile-artifact.tar.zst
# Run the tests.
- name: Run tests
env: ${{ fromJSON(inputs.envs) }}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org