You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/05/29 01:35:27 UTC
(spark) branch master updated: [SPARK-57075][INFRA] Share precompile Coursier cache with host-runner SBT jobs
This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 299e51784b97 [SPARK-57075][INFRA] Share precompile Coursier cache with host-runner SBT jobs
299e51784b97 is described below
commit 299e51784b97c1ca99d2af6373c5c50e5bd4b183
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Fri May 29 09:32:47 2026 +0800
[SPARK-57075][INFRA] Share precompile Coursier cache with host-runner SBT jobs
### What changes were proposed in this pull request?
Add `precompile-coursier-<hash>` and `precompile-coursier-` as restore-key fallbacks on `Cache Coursier local repository` for the four host-runner SBT jobs: `build` (Scala tests), `tpcds-1g`, `docker-integration-tests`, `k8s-integration-tests`. The primary key and existing prefix fallback are unchanged — the new entries are pure fallback.
### Why are the changes needed?
The `precompile` job already resolves all dependencies and writes them to `~/.cache/coursier`, but it saves under the key prefix `precompile-coursier-`, while the downstream test jobs read from `<matrix.java>-<matrix.hadoop>-coursier-`, `tpcds-coursier-`, etc. So on cold caches (new branch, modified `pom.xml` / `plugins.sbt`), the downstream jobs re-download dependencies that the precompile job already resolved minutes earlier in the same workflow.
### Does this PR introduce _any_ user-facing change?
No. CI-only.
### How was this patch tested?
Verified on https://github.com/zhengruifeng/spark/actions/runs/26564295518 (commit `919909f5cb9`): 12/12 host-runner SBT jobs restored a Coursier cache without re-downloading from Maven Central. 9 of the 12 fell back to the new `precompile-coursier-<hash>` cache; the other 3 found their own per-job cache from a prior run.
Container jobs (`pyspark`, `sparkr`, `lint`, `docs`) are excluded — their Coursier cache step is a no-op due to a host↔container `$HOME` path mismatch (host writes `/home/runner/.cache/coursier`, container looks at `/github/home/.cache/coursier`). Independent of this PR; can be addressed in a follow-up.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.7)
Closes #56118 from zhengruifeng/share-precompile-coursier-cache-dev6.
Authored-by: Ruifeng Zheng <ru...@apache.org>
Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
.github/workflows/build_and_test.yml | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml
index ee574d318fb9..66c3f89ebbab 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -385,6 +385,8 @@ jobs:
key: ${{ matrix.java }}-${{ matrix.hadoop }}-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ precompile-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
@@ -1438,6 +1440,8 @@ jobs:
key: tpcds-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
tpcds-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ precompile-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v5
with:
@@ -1556,6 +1560,8 @@ jobs:
key: docker-integration-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
docker-integration-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ precompile-coursier-
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v5
with:
@@ -1642,6 +1648,8 @@ jobs:
key: k8s-integration-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
restore-keys: |
k8s-integration-coursier-
+ precompile-coursier-${{ hashFiles('**/pom.xml', '**/plugins.sbt') }}
+ precompile-coursier-
- name: Free up disk space
run: |
if [ -f ./dev/free_disk_space ]; then
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org