You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/06/04 13:48:00 UTC
(spark) branch master updated: [SPARK-57144][INFRA] Unify Coursier cache to a single key across all jobs
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 4dbc9d7d2ec3 [SPARK-57144][INFRA] Unify Coursier cache to a single key across all jobs
4dbc9d7d2ec3 is described below
commit 4dbc9d7d2ec38b5fc09fbe8e02ec04a133bffb50
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Thu Jun 4 21:47:44 2026 +0800
[SPARK-57144][INFRA] Unify Coursier cache to a single key across all jobs
### What changes were proposed in this pull request?
Replace 8 distinct per-job Coursier cache keys with a single `coursier-<hash>` key in `.github/workflows/build_and_test.yml` and `python_hosted_runner_test.yml`:
- **`precompile`** and **`build`** (Scala test matrix): `actions/cachev5` — both can write `coursier-<hash>`. `precompile` is the primary writer (runs first, full dependency superset via all `-P` profiles). `build` is the fallback writer — when `precompile` is absent or its save fails, the first `build` matrix entry seeds the cache. When `precompile` did save it, `build` gets an exact key hit and GHA automatically skips the post-save (caches are immutable).
- **All other consumers** (`pyspark` ×9, `sparkr`, `lint`, `docs`, `tpcds-1g`, `docker-integration-tests`, `k8s-integration-tests`): converted to `actions/cache/restorev5` — restore-only, never write. `tpcds-1g` in particular only fires when SQL code changes and is skipped on the vast majority of runs, so its own Coursier cache entry would typically be LRU-evicted before the next run anyway.
### Why are the changes needed?
**1. Same-commit duplicates — ~0.01% apart by bytes.**
Per-job keys let every consumer job re-save its own copy of effectively the same content. Measured on master:
```
precompile-coursier-4f7e6f95 1,469,711,354 B current superset
25-hadoop3-coursier-4f7e6f95 1,469,562,712 B 145 KB different ← 0.01% apart
precompile-coursier-03ca361a 1,624,890,072 B previous-hash superset (~90% same)
────────────────────────────────────────────────────────────────────
total: ~4.56 GB distinct content: ~1.47 GB
```
The 145 KB delta exists because Coursier doesn't prune: on a cold run the test-matrix job restores the precompile superset via restore-key, runs tests (which resolve nothing beyond it), and its post-step re-saves a byte-for-byte copy under its own key. The per-module keys are not holding different dependency sets — they are holding copies of the same superset.
**2. Repo-wide 10 GB budget consumed by duplicates.**
Duplicates from just two branches left no room for any other branch:
```
branch-4.x: tpcds-coursier 1895 MB
21-hadoop3-coursier 1437 MB
docker-integration-coursier 1437 MB → 4770 MB
master: precompile-coursier (hash A) 1549 MB
precompile-coursier (hash B) 1401 MB
25-hadoop3-coursier (hash B) 1401 MB → 4351 MB
total: ~9.1 GB used, 10 GB budget
```
Old maintenance branches (branch-4.0, 4.1, 4.2, 3.5) had their caches evicted before their next scheduled CI run and were always cold.
**3. Dep-upgrade burst amplifies the problem.**
`pom.xml`/`plugins.sbt` are touched ~5–6 times per month on average, but upgrades cluster: on 2026-05-28 alone, 5 dependency upgrades merged in a single day (rocksdbjni, joda-time, gson, Jetty, zstd-jni). Each commit rolls the hash, so 5 consecutive CI runs each start with a cold Coursier cache. Under the old design each cold run raced to create ~5 new ~1.4 GB entries (~7 GB), immediately overflowing the budget and evicting the previous run's still-warm caches. Under the new design ea [...]
**Summary:** with one writer per branch the per-branch footprint drops from ~4.5 GB to ~1.4 GB, fitting ~6 branches in the 10 GB budget simultaneously, and a burst of dep-upgrade commits no longer triggers a cascade of mutual evictions.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
YAML validates with `python3 -c "import yaml; yaml.safe_load(...)"`.
The correctness of the one-writer design relies on two GHA cache guarantees verified in prior CI runs:
1. Caches are immutable — an exact key hit skips the post-save step (`Cache hit occurred on the primary key …, not saving cache`), so multiple jobs using `actions/cachev5` with the same key don't produce duplicates when the cache already exists.
2. The `precompile` job builds with every profile (`-Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud -Phive -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver -Pdocker-integration-tests -Pvolcano`), so its `~/.cache/coursier` is a superset of every consumer job's closure.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-sonnet-4-6)
Closes #56201 from zhengruifeng/unify-coursier-ci-cache-opt-dev6.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
.github/workflows/build_and_test.yml | 77 +++++++++++--------------
.github/workflows/python_hosted_runner_test.yml | 4 +-
2 files changed, 37 insertions(+), 44 deletions(-)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index cda9636a92e4..b36655c390f0 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -381,15 +381,14 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
- precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
- precompile-coursier-
+ ${{ runner.os }}-${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
@@ -620,9 +619,9 @@ jobs:
uses: actions/cache@v5
with:
path: ~/.cache/coursier
- key: precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- precompile-coursier-
+ ${{ runner.os }}-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v5
with:
@@ -745,13 +744,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- pyspark-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
shell: 'script -q -e -c "bash {0}"'
run: ./dev/free_disk_space_container
@@ -936,13 +935,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: sparkr-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- sparkr-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
run: ./dev/free_disk_space_container
- name: Install Java ${{ inputs.java }}
@@ -1087,13 +1086,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: docs-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- docs-coursier-
+ ${{ runner.os }}-coursier-
- name: Cache Maven local repository
uses: actions/cache@v5
with:
@@ -1286,13 +1285,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: docs-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- docs-coursier-
+ ${{ runner.os }}-coursier-
- name: Cache Maven local repository
uses: actions/cache@v5
with:
@@ -1481,15 +1480,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: tpcds-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- tpcds-coursier-
- precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
- precompile-coursier-
+ ${{ runner.os }}-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v5
with:
@@ -1615,15 +1612,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: docker-integration-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- docker-integration-coursier-
- precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
- precompile-coursier-
+ ${{ runner.os }}-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v5
with:
@@ -1703,15 +1698,13 @@ jobs:
key: build-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/mvn', 'build/sbt', 'build/sbt-launch-lib.bash', 'build/spark-build-info') }}
restore-keys: |
build-
- - name: Cache Coursier local repository
- uses: actions/cache@v5
+ - name: Restore Coursier local repository
+ uses: actions/cache/restore@v5
with:
path: ~/.cache/coursier
- key: k8s-integration-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- k8s-integration-coursier-
- precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
- precompile-coursier-
+ ${{ runner.os }}-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
diff --git a/.github/workflows/python_hosted_runner_test.yml b/.github/workflows/python_hosted_runner_test.yml
index eb0430bfe6c2..e29b89708bea 100644
--- a/.github/workflows/python_hosted_runner_test.yml
+++ b/.github/workflows/python_hosted_runner_test.yml
@@ -137,9 +137,9 @@ jobs:
uses: actions/cache@v5
with:
path: ~/.cache/coursier
- key: pyspark-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ key: ${{ runner.os }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
- pyspark-coursier-
+ ${{ runner.os }}-coursier-
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v5
with:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org