You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/15 00:06:57 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-5930-39da85081b140dae57492f96e3478337ee345a8f created (now 3ad6fe35a3)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-5930-39da85081b140dae57492f96e3478337ee345a8f
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at 3ad6fe35a3 chore: run only the cache-writing jobs on push to main (#5930)
This branch includes the following new commits:
new 3ad6fe35a3 chore: run only the cache-writing jobs on push to main (#5930)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: chore: run only the cache-writing jobs on push to main (#5930)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-5930-39da85081b140dae57492f96e3478337ee345a8f
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 3ad6fe35a30f2fad26d5757b939e66f939a30455
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Sep 15 00:06:34 2026 +0000
chore: run only the cache-writing jobs on push to main (#5930)
`pr_build_linux` runs on push to main solely to keep main's actions/cache
entries warm, because a pull request can only restore caches saved on its own
branch or on main and the merge queue's `gh-readonly-queue/*` branch takes its
caches with it. It has been doing that by running the whole pipeline: 587
runner-minutes a push, of which 514 is lints, the 5x4 linux-test matrix and the
TPC-H/TPC-DS query passes that the queue already ran against the exact tree
that landed.
Add a `cache-refresh-only` input to pr_build_linux.yml, set by ci.yml on push,
that reduces the workflow to the four jobs owning a cache entry plus the 40
second `lint` they depend on, and skips the TPC query passes while keeping
dataset generation. Routing is the existing spark_4_1 / spark_4_1_hive shape: a
second POLICY output, `build_linux_full`, with tiers ["pr", "queue"].
check-ci-config.py gains a sixth invariant so a job added to that workflow
without the guard, or a caller that drops the input, fails preflight instead of
silently putting the full pipeline back on the push tier.
Fixes #5929
---
.github/workflows/README.md | 27 +++++++--
.github/workflows/ci.yml | 13 ++++-
.github/workflows/pr_build_linux.yml | 60 ++++++++++++++++++--
dev/ci/check-ci-config.py | 107 +++++++++++++++++++++++++++++++++--
dev/ci/compute-changes.py | 24 +++++++-
docs/source/contributor-guide/ci.md | 13 ++++-
6 files changed, 224 insertions(+), 20 deletions(-)
diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index 7d591f8a99..81085bc618 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -41,8 +41,20 @@ and has to run after the commit is on main, and `pr_build_linux`, because of
`actions/cache` scoping. A pull request can only restore caches saved on its
own branch or on `main`, and the queue runs on a throwaway
`gh-readonly-queue/*` branch whose caches are deleted with it. Without a push
-run, a `Cargo.lock` or `pom.xml` change would leave the cargo-registry, Maven
-and TPC-H/TPC-DS caches on `main` stale until the next unrelated change.
+run, a `Cargo.lock` or `pom.xml` change would leave the cargo-ci, cargo-debug,
+Maven and TPC-H/TPC-DS caches on `main` stale until the next unrelated change.
+
+Warming those caches is the only thing the push run is for, so on `push` the
+Linux build runs in **cache-refresh-only** mode: `build-native`,
+`linux-test-rust` and the two TPC-H/TPC-DS jobs, each stopping once its cache
+entry is written, and nothing else. The lints, the 5x4 `linux-test` matrix and
+the TPC query runs are skipped, which takes the push tier from 587
+runner-minutes to about 73. Two POLICY outputs express this: `build_linux`
+says whether the workflow runs at all, `build_linux_full` whether it runs the
+lints and tests too, and `ci.yml` folds the second into the workflow's
+`cache-refresh-only` input. `dev/ci/check-ci-config.py` fails if a job is added
+to `pr_build_linux.yml` without either the guard or an entry in
+`CACHE_REFRESH_JOBS` naming the cache it writes. See issue #5929.
```
pull_request | merge_group | push to main | workflow_dispatch
@@ -68,7 +80,7 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change.
v v v
PR + queue tier push to main only queue tier, or PR with label
--------------- ----------------- ---------------------------
- pr_build_linux (+ push, for cache) docs pr_build_macos run-macos-tests
+ pr_build_linux (+ push, cache only) docs pr_build_macos run-macos-tests
spark_4_1 (catalyst + sql_core) pr_benchmark_check run-benchmark-check
iceberg_1_11 spark_4_1 sql_hive run-spark-4.1-hive-tests
spark_3_5 run-spark-3.5-tests
@@ -99,7 +111,7 @@ and TPC-H/TPC-DS caches on `main` stale until the next unrelated change.
| -------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| `preflight` | every PR / merge group / push / dispatch / label | none (always runs) |
| `changes` | every PR / merge group / push / dispatch / label | runs `dev/ci/compute-changes.py` |
-| `pr_build_linux` | PR, merge group or push to main, paths matched | `dev/ci/compute-changes.py` |
+| `pr_build_linux` | PR, merge group or push to main, paths matched; on push only the cache-writing jobs, via `build_linux_full` | `dev/ci/compute-changes.py` |
| `pr_build_macos` | merge group, **or** PR with `run-macos-tests` | `dev/ci/compute-changes.py` |
| `pr_benchmark_check` | merge group, **or** PR with `run-benchmark-check` | benchmark sources only |
| `docs` | push to main, paths matched | `.asf.yaml`, `docs/**`, `docs.yaml` |
@@ -200,6 +212,13 @@ in `dev/ci/compute-changes.py`:
Moving a suite between the PR and queue tiers is a one-word edit to `POLICY`.
+An output does not have to map one-to-one onto a job. Two outputs can feed a
+single call when part of a workflow belongs in a different tier from the rest:
+`spark_4_1` / `spark_4_1_hive` select which module shards the one Spark 4.1
+build runs, and `build_linux` / `build_linux_full` select whether the Linux
+build runs everything or only the jobs that populate `main`'s caches. Both
+share their `FILTERS` list by assignment so the two entries cannot drift.
+
So adding a suite, moving sources, or changing when something runs is an edit
to one of those two tables, not to ten `${{ }}` expressions. Keeping the policy
in Python is also what makes it testable: GitHub expressions cannot be
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c82ee4fbd3..e5c9d712f2 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -38,7 +38,9 @@
# push to main would double the cost of every merge. Two exceptions: `docs`
# deploys to asf-site and must run after the commit is on main, and the Linux
# build runs on push so that main's actions/cache entries stay fresh (caches
-# saved on the queue's throwaway branch are deleted with it).
+# saved on the queue's throwaway branch are deleted with it). That second
+# exception runs in `cache-refresh-only` mode: the cache writers and nothing
+# else, because the queue has already tested the tree that landed.
name: CI
@@ -157,6 +159,7 @@ jobs:
runs-on: ubuntu-slim
outputs:
build_linux: ${{ steps.compute.outputs.build_linux }}
+ build_linux_full: ${{ steps.compute.outputs.build_linux_full }}
build_macos: ${{ steps.compute.outputs.build_macos }}
benchmark: ${{ steps.compute.outputs.benchmark }}
docs: ${{ steps.compute.outputs.docs }}
@@ -233,6 +236,14 @@ jobs:
needs: changes
if: needs.changes.outputs.build_linux == 'true'
uses: ./.github/workflows/pr_build_linux.yml
+ # Two POLICY outputs feed one call, the same shape as spark_4_1 below.
+ # `build_linux` decides whether the workflow runs at all; `build_linux_full`
+ # decides whether it runs the lints and the 5x4 test matrix as well as the
+ # jobs that populate main's actions/cache entries. Only push to main sets
+ # the first without the second, which is the whole point: the queue has
+ # already tested that tree, so the push run is there for the caches alone.
+ with:
+ cache-refresh-only: ${{ needs.changes.outputs.build_linux_full != 'true' }}
pr_build_macos:
name: PR Build (macOS)
diff --git a/.github/workflows/pr_build_linux.yml b/.github/workflows/pr_build_linux.yml
index 31882d5204..8b15d829aa 100644
--- a/.github/workflows/pr_build_linux.yml
+++ b/.github/workflows/pr_build_linux.yml
@@ -19,8 +19,33 @@ name: PR Build (Linux)
# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
# live in the umbrella workflow.
+#
+# Two modes, selected by `cache-refresh-only`:
+#
+# false the full pipeline: the lints, the 5x4 linux-test matrix, the Rust
+# tests and the TPC-H/TPC-DS correctness runs. Pull requests and the
+# merge queue.
+# true only the jobs that write an actions/cache entry, and each only far
+# enough to populate it. This is the push-to-main tier, where the
+# queue has already tested the exact tree that landed and the sole
+# remaining purpose of the run is to leave main's caches warm for the
+# next pull request. See the `build_linux` / `build_linux_full`
+# comments in dev/ci/compute-changes.py for why that tier exists.
+#
+# Keeping both modes in this file is deliberate. The cache keys are the whole
+# point of the push tier, and a separate cache-warming workflow would have to
+# repeat every one of them; here they cannot drift apart. dev/ci/check-ci-config.py
+# pins which jobs survive `cache-refresh-only`, so a job added without the guard
+# cannot quietly put the full pipeline back on the push tier.
on:
workflow_call:
+ inputs:
+ cache-refresh-only:
+ description: >-
+ Populate main's actions/cache entries without running any lint or
+ test. Set by ci.yml on push to main.
+ type: boolean
+ default: false
env:
RUST_VERSION: stable
@@ -32,7 +57,9 @@ env:
jobs:
- # Fast lint check - gates all other jobs
+ # Fast lint check - gates all other jobs. Runs in cache-refresh-only mode too:
+ # it is 40 seconds, and `build-native` and `linux-test-rust` both `needs:` it,
+ # so skipping it would skip the two jobs the push tier exists to run.
lint:
name: Lint
runs-on: ubuntu-24.04
@@ -54,6 +81,7 @@ jobs:
# set, including the semantic rules, still runs in lint-java.
scalafix-syntactic:
name: Lint Scala (syntactic)
+ if: ${{ !inputs.cache-refresh-only }}
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v7
@@ -94,6 +122,7 @@ jobs:
lint-java:
needs: lint
name: Lint Java (${{ matrix.profile.name }})
+ if: ${{ !inputs.cache-refresh-only }}
runs-on: ubuntu-24.04
container:
image: amd64/rust
@@ -173,6 +202,7 @@ jobs:
build-spark-4-1:
needs: lint
name: Build Spark 4.1, JDK 17
+ if: ${{ !inputs.cache-refresh-only }}
runs-on: ubuntu-24.04
container:
image: amd64/rust
@@ -204,6 +234,7 @@ jobs:
celeborn-reflection-compatibility:
needs: lint
name: Celeborn ${{ matrix.celeborn_version }} reflection compatibility
+ if: ${{ !inputs.cache-refresh-only }}
runs-on: ubuntu-24.04
container:
image: amd64/rust
@@ -247,7 +278,9 @@ jobs:
-Dsuites=org.apache.comet.shuffle.CelebornReflectionCompatibilitySuite \
-DfailIfNoTests=false
- # Build native library once and share with all test jobs
+ # Build native library once and share with all test jobs. Also the owner of
+ # main's `cargo-ci` cache entry, which is why it runs in cache-refresh-only
+ # mode: rebuilding native/target from cold costs every pull request ~20 min.
build-native:
needs: lint
name: Build Native Library
@@ -299,7 +332,12 @@ jobs:
native/target
key: ${{ runner.os }}-cargo-ci-${{ hashFiles('native/**/Cargo.lock', 'native/**/Cargo.toml') }}-${{ hashFiles('native/**/*.rs') }}
- # Run Rust tests (runs in parallel with build-native, uses debug builds)
+ # Run Rust tests (runs in parallel with build-native, uses debug builds).
+ # Owns main's `cargo-debug` cache entry, so it runs in cache-refresh-only mode
+ # as well. The tests themselves are a small slice of its runtime -- almost all
+ # of it is the compile that populates the cache -- so there is nothing to gain
+ # from a compile-only variant, and running them keeps the cache contents
+ # identical to what a pull request restores.
linux-test-rust:
needs: lint
name: ubuntu-latest/rust-test
@@ -342,6 +380,7 @@ jobs:
linux-test:
needs: build-native
+ if: ${{ !inputs.cache-refresh-only }}
strategy:
matrix:
# the goal with these profiles is to get coverage of all Java, Scala, and Spark
@@ -552,7 +591,11 @@ jobs:
upload-test-reports: true
skip-native-build: true
- # TPC-H correctness test - verifies benchmark queries produce correct results
+ # TPC-H correctness test - verifies benchmark queries produce correct results.
+ # Runs in cache-refresh-only mode for its two cache entries: the SF=1 dataset,
+ # which is expensive to regenerate, and main's shared `java-maven` entry. Only
+ # the query run is skipped there; everything up to and including data
+ # generation still has to happen for the dataset cache to exist.
verify-benchmark-results-tpch:
needs: build-native
name: Verify TPC-H Results
@@ -608,11 +651,15 @@ jobs:
cd spark && MAVEN_OPTS='-Xmx20g' ../mvnw -B -Prelease exec:java -Dexec.mainClass="org.apache.spark.sql.GenTPCHData" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="--location `pwd`/.. --scaleFactor 1 --numPartitions 1 --overwrite"
- name: Run TPC-H queries
+ if: ${{ !inputs.cache-refresh-only }}
run: |
SPARK_HOME=`pwd` SPARK_TPCH_DATA=`pwd`/tpch/sf1_parquet ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCHQuerySuite test
# TPC-DS correctness tests - verifies benchmark queries produce correct results.
- # The three join strategies run sequentially in one job so the project is built once.
+ # The three join strategies run sequentially in one job so the project is built
+ # once. Runs in cache-refresh-only mode for the same reason as the TPC-H job;
+ # there the three query passes are skipped and only the dataset generation and
+ # the Maven cache remain.
verify-benchmark-results-tpcds:
needs: build-native
name: Verify TPC-DS Results
@@ -682,6 +729,7 @@ jobs:
cd spark && MAVEN_OPTS='-Xmx20g' ../mvnw -B -Prelease exec:java -Dexec.mainClass="org.apache.spark.sql.GenTPCDSData" -Dexec.classpathScope="test" -Dexec.cleanupDaemonThreads="false" -Dexec.args="--dsdgenDir `pwd`/../tpcds-kit/tools --location `pwd`/../tpcds-sf-1 --scaleFactor 1 --numPartitions 1"
- name: Run TPC-DS queries (Sort merge join)
+ if: ${{ !inputs.cache-refresh-only }}
run: |
SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test
env:
@@ -690,6 +738,7 @@ jobs:
spark.sql.join.preferSortMergeJoin=true
- name: Run TPC-DS queries (Broadcast hash join)
+ if: ${{ !inputs.cache-refresh-only }}
run: |
SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test
env:
@@ -697,6 +746,7 @@ jobs:
spark.sql.autoBroadcastJoinThreshold=10485760
- name: Run TPC-DS queries (Shuffled hash join)
+ if: ${{ !inputs.cache-refresh-only }}
run: |
SPARK_HOME=`pwd` SPARK_TPCDS_DATA=`pwd`/tpcds-sf-1 ./mvnw -B -Prelease -Dsuites=org.apache.spark.sql.CometTPCDSQuerySuite test
env:
diff --git a/dev/ci/check-ci-config.py b/dev/ci/check-ci-config.py
index 989dbc820b..cf9ec17866 100644
--- a/dev/ci/check-ci-config.py
+++ b/dev/ci/check-ci-config.py
@@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.
-# Guards five CI invariants that are silent when broken:
+# Guards six CI invariants that are silent when broken:
#
# 1. Change-filter routing. dev/ci/compute-changes.py decides which heavy
# jobs run. A file that a job depends on but that no filter lists makes
@@ -50,6 +50,12 @@
# run only under an input or a label can carry that for a long time
# before anyone runs them.
#
+# 6. Push-tier scope. On push to main, ci.yml calls pr_build_linux.yml with
+# `cache-refresh-only`, which reduces it to the jobs that write an
+# actions/cache entry; the merge queue already tested that tree. A job
+# added to that workflow without the guard starts running on every push
+# again and nothing fails, so nothing tells you.
+#
# Run from the repository root: python3 dev/ci/check-ci-config.py
import importlib.util
@@ -81,6 +87,7 @@ AGGREGATOR_EXEMPT = {AGGREGATOR_JOB, "docs"}
# filter deletion cannot pass unnoticed.
BUILD_JOBS = {
"build_linux",
+ "build_linux_full",
"build_macos",
"spark_3_4",
"spark_3_5",
@@ -105,7 +112,7 @@ ROUTING_CASES = [
([".github/actions/upload-artifact-retry/action.yaml"], BUILD_JOBS),
([".github/actions/download-artifact-retry/action.yaml"], BUILD_JOBS),
# The Maven bootstrap composite is called only from pr_build_linux.yml.
- ([".github/actions/maven-bootstrap/action.yaml"], {"build_linux"}),
+ ([".github/actions/maven-bootstrap/action.yaml"], {"build_linux", "build_linux_full"}),
# Spot checks that the additions above did not widen unrelated routes.
(["docs/source/user-guide/overview.md"], {"docs"}),
(["native/core/benches/parquet_read.rs"], {"benchmark"}),
@@ -115,7 +122,7 @@ ROUTING_CASES = [
# where "allowed" ignores path filters. Written out longhand rather than
# derived from POLICY, so that a change to the routing has to be stated twice
# and cannot be made by accident.
-PR_TIER = {"build_linux", "spark_4_1", "iceberg_1_11"}
+PR_TIER = {"build_linux", "build_linux_full", "spark_4_1", "iceberg_1_11"}
SPARK_OPT_IN = {"spark_3_5", "spark_4_0", "spark_4_1_hive"}
# Spark 3.4 is deprecated and sits outside the queue tier entirely: a label on
# a pull request, or a workflow_dispatch, and nothing else. Keeping it in its
@@ -135,8 +142,10 @@ POLICY_CASES = [
# deprecated Spark 3.4 suite, which no longer gates a merge.
({"name": "merge_group"}, QUEUE_TIER),
# Push to main is the site deploy plus the Linux build, which is there to
- # refresh main's actions/cache entries (see POLICY). Any other test job
- # showing up here means every merge is paying for it twice.
+ # refresh main's actions/cache entries (see POLICY). `build_linux_full`
+ # must stay out: it is what turns the lints and the test matrix back on,
+ # and the queue has already run those against the tree that landed. Any
+ # other test job showing up here means every merge is paying for it twice.
({"name": "push"}, {"docs", "build_linux"}),
# A plain pull request: the PR tier only. docs must never run here, and the
# opt-in suites stay off without their label.
@@ -257,6 +266,24 @@ JOB_KEY = re.compile(r"^ ([A-Za-z0-9_-]+):\s*$")
LOCAL_ACTION_USES = re.compile(r"uses:\s*(\./\.github/actions/\S+)")
CHECKOUT_USES = re.compile(r"uses:\s*actions/checkout@")
+# pr_build_linux.yml runs in two modes; see its header. These are the jobs that
+# must survive `cache-refresh-only`, because each one writes an actions/cache
+# entry that main needs warm for the next pull request. Anything else in that
+# file has to carry the guard.
+CACHE_REFRESH_WORKFLOW = WORKFLOWS / "pr_build_linux.yml"
+CACHE_REFRESH_JOBS = {
+ "lint": "gates build-native and linux-test-rust, and costs 40 seconds",
+ "build-native": "writes the cargo-ci cache (native/target, CI profile)",
+ "linux-test-rust": "writes the cargo-debug cache (native/target, debug)",
+ "verify-benchmark-results-tpch": "writes the TPC-H SF=1 dataset and java-maven caches",
+ "verify-benchmark-results-tpcds": "writes the TPC-DS SF=1 dataset and java-maven caches",
+}
+# Job-level `if:` only: step-level guards inside the two verify jobs are
+# indented further, and those are expected rather than a reason to exempt the
+# whole job.
+CACHE_REFRESH_GUARD = re.compile(r"^ if:.*!\s*inputs\.cache-refresh-only")
+CACHE_REFRESH_INPUT = re.compile(r"^\s+cache-refresh-only:\s*\$\{\{")
+
def load_filters():
spec = importlib.util.spec_from_file_location("compute_changes", "dev/ci/compute-changes.py")
@@ -606,6 +633,75 @@ def check_required_checks():
return not failures
+def check_cache_refresh_scope():
+ """Every job in pr_build_linux.yml is either a cache writer or guarded.
+
+ On push to main the merge queue has already tested the exact tree that
+ landed, so the only thing left for that run to do is leave main's
+ actions/cache entries warm -- a pull request can restore caches saved on
+ its own branch or on main and nowhere else, and the queue's throwaway
+ branch takes its own with it. ci.yml therefore calls the workflow with
+ `cache-refresh-only` on push, and every job that is not a cache writer
+ has to opt out with `if: ${{ !inputs.cache-refresh-only }}`.
+
+ A job added without the guard runs on every push again. Nothing fails when
+ that happens; the runner bill just quietly goes back up by up to ~500
+ minutes a push, which is what this check exists to notice.
+ """
+ failures = []
+ jobs, guarded, job, in_jobs = [], set(), None, False
+ for line in CACHE_REFRESH_WORKFLOW.read_text(encoding="utf-8").splitlines():
+ if line.startswith("jobs:"):
+ in_jobs = True
+ continue
+ if not in_jobs or line.lstrip().startswith("#"):
+ continue
+ match = JOB_KEY.match(line)
+ if match:
+ job = match.group(1)
+ jobs.append(job)
+ continue
+ if job and CACHE_REFRESH_GUARD.match(line):
+ guarded.add(job)
+
+ for stale in sorted(set(CACHE_REFRESH_JOBS) - set(jobs)):
+ failures.append(
+ f"CACHE_REFRESH_JOBS names `{stale}`, which no longer exists in "
+ f"{CACHE_REFRESH_WORKFLOW}; drop it here, or restore the job"
+ )
+ for name in jobs:
+ if name in CACHE_REFRESH_JOBS and name in guarded:
+ failures.append(
+ f"{CACHE_REFRESH_WORKFLOW}: job `{name}` is listed in "
+ f"CACHE_REFRESH_JOBS ({CACHE_REFRESH_JOBS[name]}) but carries "
+ f"the cache-refresh-only guard, so it is skipped on push and "
+ f"the cache it owns goes stale on main"
+ )
+ if name not in CACHE_REFRESH_JOBS and name not in guarded:
+ failures.append(
+ f"{CACHE_REFRESH_WORKFLOW}: job `{name}` has no "
+ f"`if: ${{{{ !inputs.cache-refresh-only }}}}`, so it runs on "
+ f"every push to main where the merge queue has already tested "
+ f"the same tree. Add the guard, or add the job to "
+ f"CACHE_REFRESH_JOBS with the cache entry it writes"
+ )
+
+ # The guards above do nothing unless the caller actually sets the input;
+ # its default is false, so a dropped `with:` block silently restores the
+ # full pipeline on push.
+ ci = (WORKFLOWS / "ci.yml").read_text(encoding="utf-8").splitlines()
+ if not any(CACHE_REFRESH_INPUT.match(line) for line in ci):
+ failures.append(
+ "ci.yml never passes `cache-refresh-only:` to pr_build_linux.yml. "
+ "The input defaults to false, so without it every push to main runs "
+ "the full pipeline again"
+ )
+
+ for failure in failures:
+ print(f"cache refresh scope: {failure}")
+ return not failures
+
+
if __name__ == "__main__":
ok = check_change_filters()
ok = check_event_policy() and ok
@@ -613,6 +709,7 @@ if __name__ == "__main__":
ok = check_artifact_names() and ok
ok = check_local_actions_have_checkout() and ok
ok = check_required_checks() and ok
+ ok = check_cache_refresh_scope() and ok
if not ok:
sys.exit(1)
print("CI config checks passed")
diff --git a/dev/ci/compute-changes.py b/dev/ci/compute-changes.py
index cf617d2e26..a66658b917 100644
--- a/dev/ci/compute-changes.py
+++ b/dev/ci/compute-changes.py
@@ -65,6 +65,12 @@ FILTERS = {
"!spark/src/test/scala/org/apache/spark/sql/benchmark/**",
"!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
],
+ # Same inputs as build_linux: not a separate job but a second POLICY
+ # decision for the same call, selecting the full pipeline rather than the
+ # cache-populating subset. ci.yml folds it into the reusable workflow's
+ # `cache-refresh-only` input. Populated below, after the dict, so the two
+ # lists cannot drift.
+ "build_linux_full": [],
"build_macos": [
"native/**",
"common/**",
@@ -318,6 +324,7 @@ FILTERS = {
],
}
FILTERS["spark_4_1_hive"] = FILTERS["spark_4_1"]
+FILTERS["build_linux_full"] = FILTERS["build_linux"]
# Which events may run each job, independent of the path filters above.
#
@@ -342,10 +349,21 @@ POLICY = {
# own branch or on main, and nowhere else. The queue runs on a throwaway
# gh-readonly-queue/* branch, so whatever it saves is deleted with that
# branch. Without a push run, a Cargo.lock or pom.xml change would leave
- # main's cargo-registry, Maven and TPC-H/TPC-DS caches stale forever, and
- # every later pull request would pay the delta on top of the restore-keys
- # prefix match.
+ # main's cargo-ci, cargo-debug, Maven and TPC-H/TPC-DS caches stale
+ # forever, and every later pull request would pay the delta on top of the
+ # restore-keys prefix match.
+ #
+ # On push that is the *only* thing it is for. The queue already tested the
+ # exact tree that landed, so re-running the lints and the 5x4 linux-test
+ # matrix there tests nothing, and they are 514 of the 587 runner-minutes a
+ # push run costs. The split below keeps the cache writers on push and moves
+ # everything else behind `build_linux_full`.
"build_linux": ["pr", "queue", "push"],
+ # The lints and the test matrix inside pr_build_linux.yml. Deliberately no
+ # "push": ci.yml turns this output into the workflow's `cache-refresh-only`
+ # input, so dropping "push" here is what trims the push tier down to the
+ # jobs that write an actions/cache entry. See issue #5929.
+ "build_linux_full": ["pr", "queue"],
# macOS runners are the scarcest capacity we have, and the Linux build
# already covers rustfmt and the Rust/JVM compile on every PR. The label
# is for a change that touches platform-specific code.
diff --git a/docs/source/contributor-guide/ci.md b/docs/source/contributor-guide/ci.md
index c7b6c87de6..c5348a622b 100644
--- a/docs/source/contributor-guide/ci.md
+++ b/docs/source/contributor-guide/ci.md
@@ -52,6 +52,14 @@ is the Linux build, which also runs on push so that the dependency caches on `ma
pull request can only restore caches saved on its own branch or on `main`, and the queue's
temporary branch takes its caches with it when it is deleted.
+That push run is for the caches and nothing else, so it runs in **cache-refresh-only** mode: only
+the four jobs that own a cache entry (the native CI build, the Rust tests, and the two TPC-H/TPC-DS
+jobs, the last two stopping before their query passes), plus the short `Lint` job the native jobs
+depend on. The lints and the Comet test matrix are skipped, which is the difference between 587
+runner-minutes a push and about 73. If you add a job to `pr_build_linux.yml`, give it
+`if: ${{ !inputs.cache-refresh-only }}` unless it writes a cache that `main` needs;
+`dev/ci/check-ci-config.py` fails the build if you forget.
+
Spark 3.4 is the one suite in neither tier. [Spark 3.4 support is deprecated](../user-guide/latest/compatibility/spark-versions.md#spark-34),
so its Spark SQL suite no longer gates a merge. It remains available on demand: apply the
`run-spark-3.4-tests` label to run it against a pull request, or trigger `ci.yml` from the Actions
@@ -202,8 +210,9 @@ does; the macOS job differs from Linux only in the platform.
The umbrella workflow, the reusable workflows it calls, and the routing tables are checked by
`dev/ci/check-ci-config.py`, which runs in preflight. It enforces that every job feeds
`Required Checks`, that the required check name in `.asf.yaml` matches the job that publishes it,
-that artifact names are unique per producer, and that the routing policy matches its test cases.
-Run it locally before pushing a CI change:
+that artifact names are unique per producer, that the routing policy matches its test cases, and
+that every job in `pr_build_linux.yml` is either a cache writer or skipped on push. Run it locally
+before pushing a CI change:
```sh
python3 dev/ci/check-ci-config.py
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org