You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/29 20:53:30 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-6392-7c9129b540331ec6da303ea95f83705bdafa432e created (now e6f5bf55d8)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6392-7c9129b540331ec6da303ea95f83705bdafa432e
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at e6f5bf55d8 ci: stop main's cargo build cache from being evicted (#6392)
This branch includes the following new commits:
new e6f5bf55d8 ci: stop main's cargo build cache from being evicted (#6392)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: ci: stop main's cargo build cache from being evicted (#6392)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6392-7c9129b540331ec6da303ea95f83705bdafa432e
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit e6f5bf55d893daf8fb9625e302fcd832360c4397
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Sep 29 14:48:50 2026 +0000
ci: stop main's cargo build cache from being evicted (#6392)
The repository's 10 GB actions/cache budget could not hold the entries
main needs: the debug native/target from linux-test-rust (4-5 GB), a
2.5 GB setup-java Maven cache that only the docs deploy reads, the Spark
SQL and TPC Maven repositories, and cargo-ci (1.4 GB). Both cargo keys
also hashed the Rust sources, so every Rust change on main wrote a new
pair. Entries evicted each other, and about half of the merge-queue runs
compiled the native library from scratch, adding ~25 minutes to the
Spark SQL build on the critical path.
- Key cargo-ci on Cargo.lock and Cargo.toml only, everywhere it is
read, and save it only when that exact key is missing.
- Stop caching native/target for linux-test-rust. It is never the long
pole, so the cold dependency compile costs runner-minutes but no wall
time, and with nothing to write it drops out of the push run. Its
Maven restore now borrows the TPC jobs' entry.
- Drop setup-java's Maven cache from the docs deploy, which runs after
the merge and gates nothing.
Closes #6387.
---
.github/actions/rust-test/action.yaml | 22 +++-----
.github/workflows/README.md | 15 +++---
.github/workflows/docs.yaml | 7 ++-
.github/workflows/iceberg_spark_test_reusable.yml | 9 ++--
.github/workflows/pr_build_linux.yml | 61 ++++++++---------------
.github/workflows/spark_sql_test_reusable.yml | 9 ++--
.github/workflows/spark_sql_writer_tests.yml | 4 +-
dev/ci/check-ci-config.py | 3 +-
dev/ci/compute-changes.py | 2 +-
docs/source/contributor-guide/ci.md | 10 ++--
10 files changed, 58 insertions(+), 84 deletions(-)
diff --git a/.github/actions/rust-test/action.yaml b/.github/actions/rust-test/action.yaml
index 85aa31979e..01e610c5be 100644
--- a/.github/actions/rust-test/action.yaml
+++ b/.github/actions/rust-test/action.yaml
@@ -41,16 +41,19 @@ runs:
cd native
cargo install cargo-machete --version 0.7.0 && cargo machete
+ # Borrows the Maven repository the TPC-H/TPC-DS jobs keep warm on main, a
+ # superset of what `common` needs. This job no longer runs on push to main
+ # (see linux-test-rust in ../../workflows/pr_build_linux.yml), so an entry
+ # of its own would never be refreshed.
- name: Restore Maven dependencies
- id: maven-cache
uses: actions/cache/restore@v4
with:
path: |
~/.m2/repository
/root/.m2/repository
- key: ${{ runner.os }}-rust-maven-${{ hashFiles('**/pom.xml') }}
+ key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}
restore-keys: |
- ${{ runner.os }}-rust-maven-
+ ${{ runner.os }}-java-maven-
- name: Build common module (pre-requisite for Rust tests)
shell: bash
@@ -58,19 +61,6 @@ runs:
cd common
../mvnw -B clean compile -DskipTests
- # Saved only on main: an entry written from a pull request or a
- # `gh-readonly-queue/*` branch cannot be restored by any later run, and it
- # evicts main's from the shared budget. See "Large caches are written on
- # main only" in ../../workflows/README.md.
- - name: Save Maven dependencies
- if: ${{ github.ref == 'refs/heads/main' && steps.maven-cache.outputs.cache-hit != 'true' }}
- uses: actions/cache/save@v4
- with:
- path: |
- ~/.m2/repository
- /root/.m2/repository
- key: ${{ runner.os }}-rust-maven-${{ hashFiles('**/pom.xml') }}
-
- name: Install nextest
shell: bash
run: |
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index a0d78c4b0d..d70f3a1bf8 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -56,15 +56,16 @@ and has to run after the commit is on main, and `pr_build_linux`, because of
`actions/cache` scoping. A pull request can only restore caches saved on its
own branch or on `main`, and the queue runs on a throwaway
`gh-readonly-queue/*` branch whose caches are deleted with it. Without a push
-run, a `Cargo.lock` or `pom.xml` change would leave the cargo-ci, cargo-debug,
-Maven and TPC-H/TPC-DS caches on `main` stale until the next unrelated change.
+run, a `Cargo.lock` or `pom.xml` change would leave the cargo-ci, Maven and
+TPC-H/TPC-DS caches on `main` stale until the next unrelated change.
Warming those caches is the only thing the push run is for, so on `push` the
-Linux build runs in **cache-refresh-only** mode: `build-native`,
-`linux-test-rust` and the two TPC-H/TPC-DS jobs, each stopping once its cache
-entry is written, and nothing else. The lints, the `linux-test` matrix and
-the TPC query runs are skipped, which takes the push tier from 587
-runner-minutes to about 73. Three POLICY outputs express this: `build_linux`
+Linux build runs in **cache-refresh-only** mode: `build-native` and the two
+TPC-H/TPC-DS jobs, each stopping once its cache entry is written, and nothing
+else. The lints, the Rust tests, the `linux-test` matrix and the TPC query runs
+are skipped, which takes the push tier from 587 runner-minutes to about 40.
+`linux-test-rust` writes no cache: a debug `native/target` is 4-5 GB, more than
+the rest of main's entries could leave room for (issue #6387). Three POLICY outputs express this: `build_linux`
says whether the workflow runs at all, `build_linux_full` whether it runs the
lints and tests too, and `build_linux_all_profiles` whether the `linux-test`
matrix covers every Spark profile or only the default one. `ci.yml` folds the
diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml
index 991a86397c..1222dbef58 100644
--- a/.github/workflows/docs.yaml
+++ b/.github/workflows/docs.yaml
@@ -42,12 +42,17 @@ jobs:
with:
python-version: "3.10"
+ # No `cache: 'maven'`: build.sh builds four Spark profiles, so that entry
+ # ran to 2.5 GB, read by this job alone, and it pushed the entries the
+ # merge-queue builds depend on out of the repository's 10 GB budget. The
+ # site deploy runs after the merge and nothing waits on it, so fetching
+ # the Maven dependencies here is the cheaper side of that trade. See
+ # issue #6387.
- name: Setup Java
uses: actions/setup-java@v6
with:
distribution: 'temurin'
java-version: '17'
- cache: 'maven'
- name: Setup Node
uses: actions/setup-node@v7
diff --git a/.github/workflows/iceberg_spark_test_reusable.yml b/.github/workflows/iceberg_spark_test_reusable.yml
index 9b4a5ae300..4154c6feb5 100644
--- a/.github/workflows/iceberg_spark_test_reusable.yml
+++ b/.github/workflows/iceberg_spark_test_reusable.yml
@@ -82,15 +82,14 @@ jobs:
run: python3 dev/ci/check-iceberg-shards.py --github-output "$GITHUB_OUTPUT"
- name: Restore Cargo cache
+ id: cargo-cache
uses: actions/cache/restore@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
native/target
- key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
- restore-keys: |
- ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}
- name: Build native library
# Use CI profile for faster builds (no LTO) and to share cache with pr_build_linux.yml.
@@ -101,13 +100,13 @@ jobs:
- name: Save Cargo cache
uses: actions/cache/save@v6
- if: github.ref == 'refs/heads/main'
+ if: github.ref == 'refs/heads/main' && steps.cargo-cache.outputs.cache-hit != 'true'
with:
path: |
~/.cargo/registry
~/.cargo/git
native/target
- key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}
- name: Upload native library
uses: ./.github/actions/upload-artifact-retry
diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml
index 9e225bbb3e..662a73e1a3 100644
--- a/.github/workflows/pr_build_linux.yml
+++ b/.github/workflows/pr_build_linux.yml
@@ -83,8 +83,8 @@ env:
jobs:
# Fast lint check - gates all other jobs. Runs in cache-refresh-only mode too:
- # it is 40 seconds, and `build-native` and `linux-test-rust` both `needs:` it,
- # so skipping it would skip the two jobs the push tier exists to run.
+ # it is 40 seconds, and `build-native` `needs:` it, so skipping it would skip
+ # the job the push tier exists to run.
lint:
name: Lint
runs-on: ubuntu-24.04
@@ -396,6 +396,11 @@ jobs:
# Build native library once and share with all test jobs. Also the owner of
# main's `cargo-ci` cache entry, which is why it runs in cache-refresh-only
# mode: rebuilding native/target from cold costs every pull request ~20 min.
+ #
+ # The key covers the dependency set only. Keying on the Rust sources as well
+ # wrote a fresh ~1.4 GB entry on every push that touched Rust, which helped
+ # nobody (the workspace crates recompile either way) and pushed main's other
+ # entries out of the repository's 10 GB budget. See issue #6387.
build-native:
needs: lint
name: Build Native Library
@@ -411,15 +416,14 @@ jobs:
jdk-version: 17 # JDK only needed for JVM module proto generation
- name: Restore Cargo cache
+ id: cargo-cache
uses: actions/cache/restore@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
native/target
- key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
- restore-keys: |
- ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}
- name: Build native library (CI profile)
run: |
@@ -439,26 +443,29 @@ jobs:
- name: Save Cargo cache
uses: actions/cache/save@v6
- # The push run is the cache warmer (see the header); a scheduled run
- # at the same sha would only re-archive an entry that already exists.
- if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+ # The push run is the cache warmer (see the header), and it writes only
+ # when this dependency set has no entry yet. A scheduled run at the
+ # same sha would only re-archive an entry that already exists.
+ if: github.event_name == 'push' && github.ref == 'refs/heads/main' && steps.cargo-cache.outputs.cache-hit != 'true'
with:
path: |
~/.cargo/registry
~/.cargo/git
native/target
- key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}
# Run Rust tests (runs in parallel with build-native, uses debug builds).
- # Owns main's `cargo-debug` cache entry, so it runs in cache-refresh-only mode
- # as well. The tests themselves are a small slice of its runtime -- almost all
- # of it is the compile that populates the cache -- so there is nothing to gain
- # from a compile-only variant, and running them keeps the cache contents
- # identical to what a pull request restores.
+ #
+ # Deliberately uncached. A debug native/target is 4-5 GB, the largest entry
+ # the repository ever held, and it kept main's `cargo-ci` entry, which the
+ # native builds on the critical path need, from staying resident. This job is
+ # never the long pole of a run, so compiling the dependencies from scratch
+ # costs about 7 runner-minutes and no wall-clock time. With no cache to write
+ # it has nothing to do on push either. See issue #6387.
linux-test-rust:
needs: lint
name: ubuntu-latest/rust-test
- if: ${{ inputs.profiles != 'nightly' }}
+ if: ${{ !inputs.cache-refresh-only && inputs.profiles != 'nightly' }}
runs-on: ubuntu-24.04
container:
image: amd64/rust
@@ -471,33 +478,9 @@ jobs:
rust-version: ${{ env.RUST_VERSION }}
jdk-version: 17
- - name: Restore Cargo cache
- uses: actions/cache/restore@v6
- with:
- path: |
- ~/.cargo/registry
- ~/.cargo/git
- native/target
- # Note: Java version intentionally excluded - Rust target is JDK-independent
- key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
- restore-keys: |
- ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
-
- name: Rust test steps
uses: ./.github/actions/rust-test
- - name: Save Cargo cache
- uses: actions/cache/save@v6
- # The push run is the cache warmer (see the header); a scheduled run
- # at the same sha would only re-archive an entry that already exists.
- if: github.event_name == 'push' && github.ref == 'refs/heads/main'
- with:
- path: |
- ~/.cargo/registry
- ~/.cargo/git
- native/target
- key: ${{ runner.os }}-cargo-debug-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
-
linux-test:
# `lint` is already upstream via build-native; it is listed here so this
# job can read its `profile-matrix` output.
diff --git a/.github/workflows/spark_sql_test_reusable.yml b/.github/workflows/spark_sql_test_reusable.yml
index 3515d64899..3a775d3eed 100644
--- a/.github/workflows/spark_sql_test_reusable.yml
+++ b/.github/workflows/spark_sql_test_reusable.yml
@@ -98,15 +98,14 @@ jobs:
jdk-version: ${{ inputs.java }}
- name: Restore Cargo cache
+ id: cargo-cache
uses: actions/cache/restore@v6
with:
path: |
~/.cargo/registry
~/.cargo/git
native/target
- key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
- restore-keys: |
- ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}
- name: Build native library (CI profile)
run: |
@@ -117,13 +116,13 @@ jobs:
- name: Save Cargo cache
uses: actions/cache/save@v6
- if: github.ref == 'refs/heads/main'
+ if: github.ref == 'refs/heads/main' && steps.cargo-cache.outputs.cache-hit != 'true'
with:
path: |
~/.cargo/registry
~/.cargo/git
native/target
- key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}
- name: Upload native library
uses: ./.github/actions/upload-artifact-retry
diff --git a/.github/workflows/spark_sql_writer_tests.yml b/.github/workflows/spark_sql_writer_tests.yml
index c813a8b39e..d59e2af1c6 100644
--- a/.github/workflows/spark_sql_writer_tests.yml
+++ b/.github/workflows/spark_sql_writer_tests.yml
@@ -90,9 +90,7 @@ jobs:
~/.cargo/registry
~/.cargo/git
native/target
- key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
- restore-keys: |
- ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-
+ key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}
- name: Build native library (CI profile)
run: |
diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py
index deab829f32..08f412dc56 100644
--- a/dev/ci/check-ci-config.py
+++ b/dev/ci/check-ci-config.py
@@ -495,9 +495,8 @@ CHECKOUT_USES = re.compile(r"uses:\s*actions/checkout@")
# file has to carry the guard.
CACHE_REFRESH_WORKFLOW = WORKFLOWS / "pr_build_linux.yml"
CACHE_REFRESH_JOBS = {
- "lint": "gates build-native and linux-test-rust, and costs 40 seconds",
+ "lint": "gates build-native, and costs 40 seconds",
"build-native": "writes the cargo-ci cache (native/target, CI profile)",
- "linux-test-rust": "writes the cargo-debug cache (native/target, debug)",
"verify-benchmark-results-tpch": "writes the TPC-H SF=1 dataset and java-maven caches",
"verify-benchmark-results-tpcds": "writes the TPC-DS SF=1 dataset and java-maven caches",
}
diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py
index a594eddbfc..52fc82f50a 100644
--- a/dev/ci/compute-changes.py
+++ b/dev/ci/compute-changes.py
@@ -480,7 +480,7 @@ POLICY = {
# own branch or on main, and nowhere else. The queue runs on a throwaway
# gh-readonly-queue/* branch, so whatever it saves is deleted with that
# branch. Without a push run, a Cargo.lock or pom.xml change would leave
- # main's cargo-ci, cargo-debug, Maven and TPC-H/TPC-DS caches stale
+ # main's cargo-ci, Maven and TPC-H/TPC-DS caches stale
# forever, and every later pull request would pay the delta on top of the
# restore-keys prefix match.
#
diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md
index 0903be4197..418f8ca7e6 100644
--- a/docs/source/contributor-guide/ci.md
+++ b/docs/source/contributor-guide/ci.md
@@ -47,7 +47,7 @@ flowchart LR
QUEUE["Queue tier, on top of the PR tier<br>Spark SQL, Spark 4.1<br>Iceberg 1.11<br>macOS build and Comet suites<br>Benchmark check, Delta gate<br>PyArrow UDF, Spark 4.0 / 4.1 / 4.2"]
NIGHTLY["Nightly tier<br>Comet suites, Spark 3.4 / 3.5 / 4.0 / 4.2<br>Spark SQL, Spark 3.5 / 4.0 / 4.2<br>Iceberg 1.8 / 1.9 / 1.10"]
S34["Neither tier<br>Spark SQL, Spark 3.4"]
- CACHE["Cache-refresh-only mode<br>the four cache-writing jobs, plus Lint"]
+ CACHE["Cache-refresh-only mode<br>the three cache-writing jobs, plus Lint"]
```
Suite by suite:
@@ -80,10 +80,10 @@ pull request can only restore caches saved on its own branch or on `main`, and t
temporary branch takes its caches with it when it is deleted.
That push run is for the caches and nothing else, so it runs in **cache-refresh-only** mode: only
-the four jobs that own a cache entry (the native CI build, the Rust tests, and the two TPC-H/TPC-DS
-jobs, the last two stopping before their query passes), plus the short `Lint` job the native jobs
-depend on. The lints and the Comet test matrix are skipped, which is the difference between 587
-runner-minutes a push and about 73. If you add a job to `pr_build_linux.yml`, give it
+the three jobs that own a cache entry (the native CI build and the two TPC-H/TPC-DS jobs, the last
+two stopping before their query passes), plus the short `Lint` job the native build depends on. The
+lints, the Rust tests and the Comet test matrix are skipped, which is the difference between 587
+runner-minutes a push and about 40. If you add a job to `pr_build_linux.yml`, give it
`if: ${{ !inputs.cache-refresh-only }}` unless it writes a cache that `main` needs;
`dev/ci/check-ci-config.py` fails the build if you forget.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org