You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/20 19:15:59 UTC
(datafusion) branch gh-readonly-queue/main/pr-25591-79650ec0e27d1f4dc11a1eeda8c084283b7a7f25 created (now 00c8ecbead)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-25591-79650ec0e27d1f4dc11a1eeda8c084283b7a7f25
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 00c8ecbead ci: run the Rust workflow for fork pull requests on the fork (#25591)
This branch includes the following new commits:
new 00c8ecbead ci: run the Rust workflow for fork pull requests on the fork (#25591)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: ci: run the Rust workflow for fork pull requests on the fork (#25591)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-25591-79650ec0e27d1f4dc11a1eeda8c084283b7a7f25
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 00c8ecbead602ff6fb53d7501b2a7b9b812ccf14
Author: Kumar Ujjawal <uj...@gmail.com>
AuthorDate: Tue Oct 20 19:15:35 2026 +0000
ci: run the Rust workflow for fork pull requests on the fork (#25591)
## Which issue does this PR close?
Part of #25148.
## Rationale for this change
Fork pull requests run the full Rust workflow upstream. Many
contributors also run that workflow on their forks, which adds demand on
shared ASF runners.
This PR uses fork runs for feedback on pull requests to `main`. The fork
run tests the branch head. The merge queue still runs all 27 Rust jobs
on the merged result before merge.
## What changes are included in this PR?
- Upstream skips 24 Rust jobs for fork pull requests to `main`.
`check-fmt`, `check-workflow-tool-installs`, and
`examples-check-datafusion-crates` still run.
- A `Rust (fork)` check shows the fork run for the pull request's head
commit. Scheduled updates pick up late runs and reruns.
- Pull requests from upstream branches and fork pull requests to release
branches keep the full Rust workflow.
- CI and local lints validate the skip conditions, job dependencies, and
artifact consumers.
The bridge uses `pull_request_target` and the GitHub API. It never
checks out or runs pull request code. The fork check is informational
because contributors control their fork's workflows. It is not a
required check for merge.
## What is the testing strategy for this PR?
Local checks passed for the job conditions, dependencies, and artifact
guards. Bridge probes with stubbed API responses covered missing runs,
late runs, reruns, and pagination beyond 500 open pull requests.
Hosted end-to-end validation of the new bridge remains outstanding.
`pull_request_target` uses the base branch's workflow, so this pull
request cannot exercise the new bridge on GitHub.
## Are there any user-facing changes?
Contributors must enable GitHub Actions on their forks. The contributor
guide and PR template explain this requirement. Pushes that change only
documentation or Markdown do not start the Rust workflow.
---------
Co-authored-by: kosiew <ko...@gmail.com>
---
.github/pull_request_template.md | 7 +-
.github/workflows/dev.yml | 6 +
.github/workflows/fork_ci_status.yml | 319 ++++++++++++++++++++++++
.github/workflows/rust.yml | 57 +++--
ci/scripts/check_asf_yaml_status_checks.py | 164 +------------
ci/scripts/check_rust_workflow_skips.py | 284 ++++++++++++++++++++++
ci/scripts/test_fork_ci_status.cjs | 373 +++++++++++++++++++++++++++++
ci/scripts/test_rust_workflow_skips.py | 71 ++++++
dev/rust_lint.sh | 11 +-
docs/source/contributor-guide/index.md | 23 ++
docs/source/contributor-guide/testing.md | 27 ++-
11 files changed, 1155 insertions(+), 187 deletions(-)
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index f3c069f5d9..bdcabb8deb 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -1,3 +1,8 @@
+> [!IMPORTANT]
+>
+> **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push.
+> See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks).
+
## Which issue does this PR close?
<!--
@@ -10,7 +15,7 @@ We generally require a GitHub issue to be filed for all bug fixes and enhancemen
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
- Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
+ Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
Please explain the problem you are trying to solve in terms of the user-visible
behavior, rather than the implementation.
diff --git a/.github/workflows/dev.yml b/.github/workflows/dev.yml
index bee4cf5ec1..a34ef5b50f 100644
--- a/.github/workflows/dev.yml
+++ b/.github/workflows/dev.yml
@@ -83,6 +83,12 @@ jobs:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- run: pip install pyyaml
- run: python3 ci/scripts/check_asf_yaml_status_checks.py
+ - run: python3 ci/scripts/check_rust_workflow_skips.py
+ - run: python3 ci/scripts/test_rust_workflow_skips.py
+ - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
+ with:
+ node-version: "24"
+ - run: node --test ci/scripts/test_fork_ci_status.cjs
typos:
name: Spell Check with Typos
diff --git a/.github/workflows/fork_ci_status.yml b/.github/workflows/fork_ci_status.yml
new file mode 100644
index 0000000000..56213480b4
--- /dev/null
+++ b/.github/workflows/fork_ci_status.yml
@@ -0,0 +1,319 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# # Fork CI status
+#
+# A pull request from a fork to `main` runs `rust.yml` on the fork. This
+# workflow mirrors that run as one `Rust (fork)` check on the pull request, so
+# reviewers see the result without opening the fork. It never checks out or
+# runs code from the pull request: on `pull_request_target` it reads the fork's
+# public run list and creates the check, and on a schedule it updates the
+# active or missing runs of recently updated pull requests plus a rotating
+# slice of all open pull requests. The check is informational. The merge
+# queue runs the full workflow on the merged result and stays the gate.
+#
+# See docs/source/contributor-guide/index.md, "Pull Requests from Forks".
+name: Fork CI status
+
+on:
+ pull_request_target:
+ types: [opened, reopened, synchronize]
+ schedule:
+ - cron: "*/15 * * * *"
+ workflow_dispatch:
+ inputs:
+ hours:
+ description: "Update every pull request updated within this many hours"
+ type: number
+ default: 6
+
+permissions:
+ actions: read
+ checks: write
+ pull-requests: read
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || 'update' }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request_target' }}
+
+jobs:
+ bridge:
+ name: Mirror the fork run
+ runs-on: ubuntu-slim
+ timeout-minutes: 10
+ steps:
+ - name: Create or update the check
+ uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+ env:
+ UPDATE_HOURS: ${{ inputs.hours || 6 }}
+ with:
+ script: |
+ const CHECK = 'Rust (fork)';
+ const WORKFLOW = 'rust.yml';
+ const upstream = { owner: context.repo.owner, repo: context.repo.repo };
+ const GUIDE = 'https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks';
+ // Older pull requests are revisited this many at a time, so a rerun
+ // or a late run on a quiet pull request is mirrored within hours.
+ const SWEEP_SIZE = 25;
+
+ // A check run accepts a fixed set of conclusions; a workflow run can
+ // report others, such as startup_failure.
+ const CONCLUSIONS = new Set([
+ 'action_required', 'cancelled', 'failure', 'neutral',
+ 'skipped', 'stale', 'success', 'timed_out',
+ ]);
+ const conclusionOf = (run) =>
+ CONCLUSIONS.has(run.conclusion) ? run.conclusion : 'failure';
+ const statusOf = (run) => {
+ if (run.status === 'completed') return 'completed';
+ return run.status === 'in_progress' ? 'in_progress' : 'queued';
+ };
+ const isForkPrToMain = (pr) =>
+ Boolean(pr.head.repo) && pr.head.repo.fork === true && pr.base.ref === 'main';
+
+ const HOW_UPDATED =
+ 'Active or missing runs on recently updated pull requests are checked every 15 minutes. ' +
+ 'Completed runs and older pull requests are checked by a rotating sweep, which can take a few hours. ' +
+ 'The merge queue runs the full workflow on the merged result before this pull request merges.';
+
+ // The check for a run. `text` is read back by the scheduled update.
+ const runPayload = (fork, sha, run) => {
+ const status = statusOf(run);
+ return {
+ status,
+ ...(status === 'completed' ? { conclusion: conclusionOf(run) } : {}),
+ details_url: run.html_url,
+ output: {
+ title: `Fork run ${status === 'completed' ? run.conclusion : status}`,
+ summary: [
+ `[Rust workflow run on ${fork.owner}/${fork.repo}](${run.html_url}) for this commit.`,
+ '',
+ 'If the fork run failed for reasons unrelated to this pull request, rerun it on the fork.',
+ HOW_UPDATED,
+ ].join('\n'),
+ text: JSON.stringify({
+ owner: fork.owner,
+ repo: fork.repo,
+ head_sha: sha,
+ run_id: run.id,
+ run_attempt: run.run_attempt,
+ }),
+ },
+ };
+ };
+
+ // The check when the fork has no run for the commit. The workflow's
+ // state on the fork says whether workflows are enabled there; a
+ // push that changes only documentation or Markdown does not start
+ // rust.yml even when they are. The scheduled update keeps looking
+ // for a run in every case.
+ const missingPayload = (fork, sha, state) => {
+ const disabled = typeof state === 'string' && state.startsWith('disabled');
+ const active = state === 'active';
+ const advice = disabled
+ ? [
+ '1. Enable workflows on your fork: open its **Actions** tab and confirm the prompt.',
+ '2. Push again. A push that changes only documentation or Markdown does not start',
+ ' the Rust workflow.',
+ ]
+ : active
+ ? [
+ 'A push that changes only documentation or Markdown does not start the Rust',
+ 'workflow, and a run can take a moment to appear. This check keeps looking for one.',
+ ]
+ : [
+ 'GitHub reports no Rust workflow on the fork, which usually means workflows are not',
+ 'enabled there. Enable them on the fork\'s **Actions** tab and push again. A push that',
+ 'changes only documentation or Markdown does not start the Rust workflow.',
+ ];
+ return {
+ status: 'completed',
+ conclusion: disabled ? 'action_required' : active ? 'skipped' : 'neutral',
+ output: {
+ title: disabled ? 'Workflows are disabled on the fork' : 'No fork run for this commit',
+ summary: [
+ `No Rust workflow run was found on ${fork.owner}/${fork.repo} for ${sha}.`,
+ '',
+ ...advice,
+ '',
+ HOW_UPDATED,
+ '',
+ `See ${GUIDE}`,
+ ].join('\n'),
+ text: JSON.stringify({ owner: fork.owner, repo: fork.repo, head_sha: sha, run_id: null }),
+ },
+ };
+ };
+
+ // The fork creates its run a few seconds after the push.
+ async function findRun(fork, sha, attempts) {
+ for (let attempt = 0; attempt < attempts; attempt++) {
+ try {
+ const { data } = await github.rest.actions.listWorkflowRuns({
+ owner: fork.owner,
+ repo: fork.repo,
+ workflow_id: WORKFLOW,
+ head_sha: sha,
+ per_page: 10,
+ });
+ // A newer manual run can replace a cancelled push run.
+ const run = data.workflow_runs[0];
+ if (run) return run;
+ } catch (error) {
+ // 404 when the fork has Actions disabled or lacks the workflow.
+ core.info(`No run list for ${fork.owner}/${fork.repo}: ${error.message}`);
+ }
+ if (attempt + 1 < attempts) {
+ await new Promise((resolve) => setTimeout(resolve, 5000));
+ }
+ }
+ return null;
+ }
+
+ // `active`, or `disabled_fork` until the owner enables workflows;
+ // null when the workflow cannot be read.
+ async function workflowState(fork) {
+ try {
+ const { data } = await github.rest.actions.getWorkflow({
+ owner: fork.owner,
+ repo: fork.repo,
+ workflow_id: WORKFLOW,
+ });
+ return data.state || null;
+ } catch (error) {
+ core.info(`No workflow state for ${fork.owner}/${fork.repo}: ${error.message}`);
+ return null;
+ }
+ }
+
+ async function createCheck(pr, attempts) {
+ const fork = { owner: pr.head.repo.owner.login, repo: pr.head.repo.name };
+ const sha = pr.head.sha;
+ const run = await findRun(fork, sha, attempts);
+ const payload = run
+ ? runPayload(fork, sha, run)
+ : missingPayload(fork, sha, await workflowState(fork));
+ await github.rest.checks.create({ ...upstream, name: CHECK, head_sha: sha, ...payload });
+ core.info(`Created ${CHECK} for ${sha}: ${payload.conclusion || payload.status}`);
+ }
+
+ async function notify() {
+ const pr = context.payload.pull_request;
+ // Only fork PRs to main skip jobs upstream; release branches have
+ // no merge queue and keep the full run.
+ if (!isForkPrToMain(pr)) {
+ core.info('Not a fork pull request to main; nothing to mirror.');
+ return;
+ }
+ await createCheck(pr, 12);
+ }
+
+ // Discover the latest run again, since a manual run can replace the
+ // recorded run, and the fork can delete old runs.
+ async function syncCheck(pr, includeCompleted = true) {
+ const { data } = await github.rest.checks.listForRef({
+ ...upstream,
+ ref: pr.head.sha,
+ check_name: CHECK,
+ per_page: 5,
+ });
+ const check = data.check_runs[0];
+ if (!check) {
+ // Recover if the initial notification failed before creating a check.
+ await createCheck(pr, 1);
+ return;
+ }
+ if (!check.output || !check.output.text) return;
+ let params;
+ try {
+ params = JSON.parse(check.output.text);
+ } catch {
+ return;
+ }
+ // Completed runs only need the rotating sweep to discover reruns.
+ // A completed check with no run must keep looking for a late run.
+ if (!includeCompleted && check.status === 'completed' && params.run_id) return;
+ const fork = { owner: pr.head.repo.owner.login, repo: pr.head.repo.name };
+ const sha = pr.head.sha;
+ const run = await findRun(fork, sha, 1);
+ if (!run) return;
+ const payload = runPayload(fork, sha, run);
+ if (
+ payload.status === check.status &&
+ (payload.conclusion || null) === check.conclusion &&
+ run.id === params.run_id &&
+ run.run_attempt === params.run_attempt
+ ) {
+ return;
+ }
+ await github.rest.checks.update({ ...upstream, check_run_id: check.id, ...payload });
+ core.info(`#${pr.number}: ${check.status}/${check.conclusion} -> ${payload.status}/${payload.conclusion || ''}`);
+ }
+
+ const listing = { ...upstream, state: 'open', sort: 'updated', direction: 'desc' };
+
+ // The number of open pull requests, from the last-page link of a
+ // one-item page: one request, whatever the count.
+ async function openPullRequestCount() {
+ const response = await github.rest.pulls.list({ ...listing, per_page: 1 });
+ const last = /[?&]page=(\d+)>;\s*rel="last"/.exec(response.headers.link || '');
+ return last ? Number(last[1]) : response.data.length;
+ }
+
+ async function update() {
+ const hours = Number(process.env.UPDATE_HOURS) || 6;
+ const cutoff = Date.now() - hours * 3600 * 1000;
+ const seen = new Set();
+
+ // Select the sweep first so a recent completed PR in this slice
+ // is checked once, even though it also appears in the window.
+ const slices = Math.max(1, Math.ceil((await openPullRequestCount()) / SWEEP_SIZE));
+ const slot = Math.floor(Date.now() / (15 * 60 * 1000));
+ const page = (slot % slices) + 1;
+ const { data: slice } = await github.rest.pulls.list({ ...listing, per_page: SWEEP_SIZE, page });
+ const sweepNumbers = new Set(slice.map((pr) => pr.number));
+
+ // Pull requests updated within the window, newest first. Stop at
+ // the window: GITHUB_TOKEN allows 1,000 requests per hour per
+ // repository.
+ const pages = github.paginate.iterator(github.rest.pulls.list, { ...listing, per_page: 100 });
+ for await (const page of pages) {
+ let stale = false;
+ for (const pr of page.data) {
+ if (new Date(pr.updated_at).getTime() < cutoff) {
+ stale = true;
+ break;
+ }
+ seen.add(pr.number);
+ if (isForkPrToMain(pr)) await syncCheck(pr, sweepNumbers.has(pr.number));
+ }
+ if (stale) break;
+ }
+
+ // Check the rest of the slice, including older pull requests.
+ // A fork rerun does not update the upstream pull request.
+ for (const pr of slice) {
+ if (seen.has(pr.number) || !isForkPrToMain(pr)) continue;
+ await syncCheck(pr);
+ }
+ }
+
+ if (context.eventName === 'pull_request_target') {
+ await notify();
+ } else {
+ await update();
+ }
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index 383d268854..1b68535fdd 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -38,13 +38,25 @@
# The other jobs carry an `if:` that skips them for that event alone. PRs, the
# merge queue, `workflow_dispatch`, and every other branch run all jobs. A
# direct push that bypasses the queue is not re-checked; dispatch the workflow
-# on `main` for a full run. `ci/scripts/check_asf_yaml_status_checks.py` keeps
+# on `main` for a full run. `ci/scripts/check_rust_workflow_skips.py` keeps
# the skipped set explicit.
#
+# # Pull requests from forks
+#
+# A PR from a fork runs this workflow on the fork, on the contributor's own
+# runners, and `fork_ci_status.yml` mirrors that run as the `Rust (fork)` check
+# on the PR. Upstream runs `check-fmt`, `check-workflow-tool-installs`, and
+# `examples-check-datafusion-crates` for such a PR; the other jobs carry an
+# `if:` that skips them. The merge queue runs every job on the merged result,
+# so it stays the gate. PRs from branches in this repository, and fork PRs to
+# release branches, which have no merge queue, run every job here, as before.
+#
name: Rust
concurrency:
- group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }}
+ # Fork pushes cancel superseded runs on the same branch. Upstream main and
+ # merge queue runs keep separate groups for each SHA.
+ group: ${{ github.repository }}-${{ github.head_ref || (github.repository == 'apache/datafusion' && github.sha || github.ref) }}-${{ github.workflow }}
cancel-in-progress: true
on:
@@ -71,6 +83,7 @@ jobs:
# Check crate compiles and base cargo check passes
linux-build-lib:
name: linux build test
+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=8,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -121,7 +134,7 @@ jobs:
linux-datafusion-common-features:
name: cargo check datafusion-common features
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-latest
container:
image: amd64/rust
@@ -152,7 +165,7 @@ jobs:
linux-datafusion-substrait-features:
name: cargo check datafusion-substrait features
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
env:
CARGO_INCREMENTAL: "0"
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
@@ -200,7 +213,7 @@ jobs:
linux-datafusion-proto-features:
name: cargo check datafusion-proto features
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -237,6 +250,7 @@ jobs:
linux-datafusion-ffi-features:
name: cargo check datafusion-ffi features
needs: linux-build-lib
+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -263,7 +277,7 @@ jobs:
linux-cargo-check-datafusion:
name: cargo check datafusion features
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
env:
CARGO_INCREMENTAL: "0"
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
@@ -339,6 +353,7 @@ jobs:
linux-cargo-check-datafusion-functions:
name: cargo check datafusion-functions features
needs: linux-build-lib
+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-latest
container:
image: amd64/rust
@@ -384,7 +399,7 @@ jobs:
linux-cargo-check-datafusion-spark:
name: cargo check datafusion-spark features
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-latest
container:
image: amd64/rust
@@ -413,6 +428,7 @@ jobs:
linux-test:
name: cargo test (amd64)
needs: linux-build-lib
+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -466,7 +482,7 @@ jobs:
linux-test-datafusion-cli:
name: cargo test datafusion-cli (amd64)
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
steps:
- uses: runs-on/action@17b59092eb37ebfdbb034460892887c09100eaa3 # v2.5.0
@@ -512,6 +528,7 @@ jobs:
linux-test-example:
name: cargo examples (amd64)
needs: linux-build-lib
+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -543,7 +560,7 @@ jobs:
linux-test-doc:
name: cargo test doc (amd64)
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -566,7 +583,7 @@ jobs:
linux-rustdoc:
name: cargo doc
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -582,7 +599,7 @@ jobs:
linux-wasm-pack:
name: build and run with wasm-pack
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-24.04
env:
# Use the preinstalled Clang 18 even if the runner's default Clang changes.
@@ -614,7 +631,7 @@ jobs:
verify-benchmark-results:
name: verify benchmark results (amd64)
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -647,7 +664,7 @@ jobs:
sqllogictest-postgres:
name: "Run sqllogictest with Postgres runner"
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -685,7 +702,7 @@ jobs:
sqllogictest-substrait:
name: "Run sqllogictest in Substrait round-trip mode"
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -734,6 +751,7 @@ jobs:
# flaky sqllogictest metrics.
macos-aarch64:
name: cargo test (macos-aarch64)
+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: macos-15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -748,7 +766,7 @@ jobs:
vendor:
name: Verify Vendored Code
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-latest
container:
image: amd64/rust
@@ -793,6 +811,7 @@ jobs:
clippy:
name: clippy
needs: linux-build-lib
+ if: ${{ !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -820,7 +839,7 @@ jobs:
cargo-toml-formatting-checks:
name: check Cargo.toml formatting
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-latest
container:
image: amd64/rust
@@ -844,7 +863,7 @@ jobs:
config-docs-check:
name: check configs.md and ***_functions.md is up-to-date
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ${{ vars.USE_RUNS_ON == 'true' && format('runs-on={0},family=m8a+m7a+c8a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
container:
image: amd64/rust
@@ -874,7 +893,7 @@ jobs:
examples-docs-check:
name: check example README is up-to-date
needs: linux-build-lib
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-latest
container:
image: amd64/rust
@@ -917,7 +936,7 @@ jobs:
# - datafusion-cli
msrv:
name: Verify MSRV (Min Supported Rust Version)
- if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') }}
+ if: ${{ !(github.event_name == 'push' && github.ref == 'refs/heads/main' && github.repository == 'apache/datafusion') && !(github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork && github.base_ref == 'main') }}
runs-on: ubuntu-latest
container:
image: amd64/rust
diff --git a/ci/scripts/check_asf_yaml_status_checks.py b/ci/scripts/check_asf_yaml_status_checks.py
index 1bd2852f4e..1356541590 100755
--- a/ci/scripts/check_asf_yaml_status_checks.py
+++ b/ci/scripts/check_asf_yaml_status_checks.py
@@ -24,11 +24,6 @@ check from running on some PRs, blocking merges).
A typo or stale entry in required_status_checks will block all
merges for the project, so this check catches that early.
-
-Also validate the post-merge conditions in rust.yml: exactly the jobs
-in POST_MERGE_SKIPPED_JOBS carry the condition that skips them on
-pushes to upstream `main`, and the Cargo check artifact steps are
-guarded the same way.
"""
import glob
@@ -37,57 +32,6 @@ import sys
import yaml
-# Commits reach upstream `main` through the merge queue, which already ran
-# rust.yml at the same SHA. On a push to upstream `main`, rust.yml skips the
-# jobs below and keeps only the jobs that save caches, publish coverage, or run
-# the non-required FFI check. A job is skipped only when it is listed here and
-# carries exactly POST_MERGE_SKIP_CONDITION, so a new job runs by default and
-# changing the set is a deliberate edit to both places.
-RUST_WORKFLOW = "rust.yml"
-POST_MERGE_PUSH = (
- "github.event_name == 'push'"
- " && github.ref == 'refs/heads/main'"
- " && github.repository == 'apache/datafusion'"
-)
-POST_MERGE_SKIP_CONDITION = "${{ !(" + POST_MERGE_PUSH + ") }}"
-POST_MERGE_SKIPPED_JOBS = frozenset(
- {
- "cargo-toml-formatting-checks",
- "check-fmt",
- "check-workflow-tool-installs",
- "config-docs-check",
- "examples-docs-check",
- "linux-cargo-check-datafusion",
- "linux-cargo-check-datafusion-spark",
- "linux-datafusion-common-features",
- "linux-datafusion-proto-features",
- "linux-datafusion-substrait-features",
- "linux-rustdoc",
- "linux-test-datafusion-cli",
- "linux-test-doc",
- "linux-wasm-pack",
- "msrv",
- "sqllogictest-postgres",
- "sqllogictest-substrait",
- "vendor",
- "verify-benchmark-results",
- }
-)
-# `linux-build-lib` uploads its Cargo check artifacts for two skipped jobs, so
-# the archive and upload steps skip on the same pushes.
-ARTIFACT_JOB = "linux-build-lib"
-ARTIFACT_ARCHIVE_STEP = "archive-check"
-ARTIFACT_NAME = "cargo-check"
-ARTIFACT_UPLOAD_CONDITION = (
- "${{ steps." + ARTIFACT_ARCHIVE_STEP + ".outcome == 'success'"
- " && !(" + POST_MERGE_PUSH + ") }}"
-)
-
-
-def load_workflow(path):
- with open(path) as f:
- return yaml.safe_load(f)
-
def get_required_checks(asf_yaml_path):
"""Extract all required_status_checks contexts from .asf.yaml."""
@@ -114,7 +58,8 @@ def get_workflow_jobs(workflows_dir):
"""
jobs = {} # identifier -> [(workflow_file, has_path_filters)]
for workflow_file in sorted(glob.glob(os.path.join(workflows_dir, "*.yml"))):
- workflow = load_workflow(workflow_file)
+ with open(workflow_file) as f:
+ workflow = yaml.safe_load(f)
if not workflow or "jobs" not in workflow:
continue
@@ -140,90 +85,6 @@ def get_workflow_jobs(workflows_dir):
return jobs
-def iter_steps(job_config):
- """Yield the steps of a job, flattening `parallel:` groups."""
- for step in job_config.get("steps") or []:
- if not isinstance(step, dict):
- continue
- if "parallel" in step:
- yield from (s for s in step["parallel"] or [] if isinstance(s, dict))
- else:
- yield step
-
-
-def uses_action(step, action):
- return str(step.get("uses", "")).startswith(action)
-
-
-def check_post_merge_conditions(workflow):
- """Return the problems with rust.yml's post-merge job and artifact conditions."""
- errors = []
- jobs = {
- key: config
- for key, config in (workflow.get("jobs") or {}).items()
- if isinstance(config, dict)
- }
-
- for job in sorted(POST_MERGE_SKIPPED_JOBS - set(jobs)):
- errors.append(
- f" - job `{job}` is listed in POST_MERGE_SKIPPED_JOBS "
- f"but does not exist in {RUST_WORKFLOW}"
- )
- for job, config in sorted(jobs.items()):
- condition = config.get("if")
- if job in POST_MERGE_SKIPPED_JOBS:
- if condition != POST_MERGE_SKIP_CONDITION:
- errors.append(
- f" - job `{job}` must carry exactly "
- f"`if: {POST_MERGE_SKIP_CONDITION}`, found {condition!r}"
- )
- elif condition is not None and any(
- token in str(condition)
- for token in ("github.event_name", "github.ref", "refs/heads/")
- ):
- errors.append(
- f" - job `{job}` is not listed in POST_MERGE_SKIPPED_JOBS "
- f"but carries an event condition: {condition!r}"
- )
-
- build = jobs.get(ARTIFACT_JOB)
- if build is None:
- errors.append(f" - job `{ARTIFACT_JOB}` does not exist in {RUST_WORKFLOW}")
- else:
- steps = list(iter_steps(build))
- archive = [s for s in steps if s.get("id") == ARTIFACT_ARCHIVE_STEP]
- uploads = [s for s in steps if uses_action(s, "actions/upload-artifact")]
- if len(archive) != 1 or archive[0].get("if") != POST_MERGE_SKIP_CONDITION:
- found = [s.get("if") for s in archive]
- errors.append(
- f" - the step `id: {ARTIFACT_ARCHIVE_STEP}` in `{ARTIFACT_JOB}` "
- f"must exist once and carry exactly "
- f"`if: {POST_MERGE_SKIP_CONDITION}`, found {found!r}"
- )
- if len(uploads) != 1 or uploads[0].get("if") != ARTIFACT_UPLOAD_CONDITION:
- found = [s.get("if") for s in uploads]
- errors.append(
- f" - the upload-artifact step in `{ARTIFACT_JOB}` must exist "
- f"once and carry exactly `if: {ARTIFACT_UPLOAD_CONDITION}`, "
- f"found {found!r}"
- )
-
- for job, config in sorted(jobs.items()):
- downloads = [
- s
- for s in iter_steps(config)
- if uses_action(s, "actions/download-artifact")
- and (s.get("with") or {}).get("name") == ARTIFACT_NAME
- ]
- if downloads and job not in POST_MERGE_SKIPPED_JOBS:
- errors.append(
- f" - job `{job}` downloads the `{ARTIFACT_NAME}` artifact, which "
- f"is not uploaded on pushes to main, so it must be listed in "
- f"POST_MERGE_SKIPPED_JOBS"
- )
- return errors
-
-
def main():
repo_root = os.path.dirname(
os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
@@ -234,6 +95,7 @@ def main():
required_checks = get_required_checks(asf_yaml)
if not required_checks:
print("No required_status_checks found in .asf.yaml — nothing to validate.")
+ return
jobs = get_workflow_jobs(workflows_dir)
errors = []
@@ -264,10 +126,6 @@ def main():
f"and will block merging"
)
- post_merge_errors = check_post_merge_conditions(
- load_workflow(os.path.join(workflows_dir, RUST_WORKFLOW))
- )
-
if errors:
print("ERROR: Problems found with required_status_checks in .asf.yaml:\n")
print("\n".join(errors))
@@ -275,23 +133,11 @@ def main():
print("Available job names across all workflows:")
for name in sorted(jobs):
print(f" - {name}")
- if post_merge_errors:
- print(
- f"ERROR: Problems found with post-merge conditions in {RUST_WORKFLOW}:\n"
- )
- print("\n".join(post_merge_errors))
- print()
- if errors or post_merge_errors:
sys.exit(1)
- if required_checks:
- print(
- f"OK: All {len(required_checks)} required_status_checks "
- "match existing GitHub Actions jobs."
- )
print(
- f"OK: {RUST_WORKFLOW} skips exactly {len(POST_MERGE_SKIPPED_JOBS)} "
- "jobs on pushes to main."
+ f"OK: All {len(required_checks)} required_status_checks "
+ "match existing GitHub Actions jobs."
)
diff --git a/ci/scripts/check_rust_workflow_skips.py b/ci/scripts/check_rust_workflow_skips.py
new file mode 100755
index 0000000000..4d9ea090d3
--- /dev/null
+++ b/ci/scripts/check_rust_workflow_skips.py
@@ -0,0 +1,284 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Validate the job-level skip conditions in rust.yml.
+
+rust.yml skips jobs on two events. Each skip set below pairs a clause with
+the jobs that carry it: a job's `if:` is exactly `!(clause)` for every set
+it belongs to, joined with ` && ` in the order of SKIP_POLICIES, and a job
+in no set carries no job-level `if:`. A new job therefore runs everywhere
+by default, and changing a set is a deliberate edit to both files.
+
+The Cargo check artifact steps in `linux-build-lib` and the jobs that
+download the artifact must follow the same sets.
+"""
+
+import os
+import sys
+
+import yaml
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+from check_asf_yaml_status_checks import get_required_checks # noqa: E402
+
+RUST_WORKFLOW = "rust.yml"
+
+# Skipped on a push to upstream main; see "Pushes to main" in rust.yml.
+POST_MERGE_PUSH = (
+ "github.event_name == 'push'"
+ " && github.ref == 'refs/heads/main'"
+ " && github.repository == 'apache/datafusion'"
+)
+POST_MERGE_SKIPPED_JOBS = frozenset(
+ {
+ "cargo-toml-formatting-checks",
+ "check-fmt",
+ "check-workflow-tool-installs",
+ "config-docs-check",
+ "examples-docs-check",
+ "linux-cargo-check-datafusion",
+ "linux-cargo-check-datafusion-spark",
+ "linux-datafusion-common-features",
+ "linux-datafusion-proto-features",
+ "linux-datafusion-substrait-features",
+ "linux-rustdoc",
+ "linux-test-datafusion-cli",
+ "linux-test-doc",
+ "linux-wasm-pack",
+ "msrv",
+ "sqllogictest-postgres",
+ "sqllogictest-substrait",
+ "vendor",
+ "verify-benchmark-results",
+ }
+)
+
+# Skipped on a pull request from a fork; see "Pull requests from forks" in
+# rust.yml and fork_ci_status.yml.
+FORK_PULL_REQUEST = (
+ "github.event_name == 'pull_request'"
+ " && github.event.pull_request.head.repo.fork"
+ " && github.base_ref == 'main'"
+)
+FORK_PR_SKIPPED_JOBS = frozenset(
+ {
+ "cargo-toml-formatting-checks",
+ "clippy",
+ "config-docs-check",
+ "examples-docs-check",
+ "linux-build-lib",
+ "linux-cargo-check-datafusion",
+ "linux-cargo-check-datafusion-functions",
+ "linux-cargo-check-datafusion-spark",
+ "linux-datafusion-common-features",
+ "linux-datafusion-ffi-features",
+ "linux-datafusion-proto-features",
+ "linux-datafusion-substrait-features",
+ "linux-rustdoc",
+ "linux-test",
+ "linux-test-datafusion-cli",
+ "linux-test-doc",
+ "linux-test-example",
+ "linux-wasm-pack",
+ "macos-aarch64",
+ "msrv",
+ "sqllogictest-postgres",
+ "sqllogictest-substrait",
+ "vendor",
+ "verify-benchmark-results",
+ }
+)
+
+SKIP_POLICIES = (
+ ("POST_MERGE_SKIPPED_JOBS", POST_MERGE_SKIPPED_JOBS, POST_MERGE_PUSH),
+ ("FORK_PR_SKIPPED_JOBS", FORK_PR_SKIPPED_JOBS, FORK_PULL_REQUEST),
+)
+
+
+def expected_condition(job):
+ """The exact job-level `if:` for a job, or None when it must carry none."""
+ clauses = [clause for _, jobs, clause in SKIP_POLICIES if job in jobs]
+ if not clauses:
+ return None
+ return "${{ " + " && ".join(f"!({clause})" for clause in clauses) + " }}"
+
+
+# `linux-build-lib` uploads Cargo check artifacts for two consumers. Its steps
+# skip on a push to main and the whole job skips on a fork PR.
+ARTIFACT_JOB = "linux-build-lib"
+ARTIFACT_ARCHIVE_STEP = "archive-check"
+ARTIFACT_NAME = "cargo-check"
+ARTIFACT_ARCHIVE_CONDITION = "${{ !(" + POST_MERGE_PUSH + ") }}"
+ARTIFACT_UPLOAD_CONDITION = (
+ "${{ steps." + ARTIFACT_ARCHIVE_STEP + ".outcome == 'success'"
+ " && !(" + POST_MERGE_PUSH + ") }}"
+)
+
+
+def iter_steps(job_config):
+ """Yield the steps of a job, flattening `parallel:` groups."""
+ for step in job_config.get("steps") or []:
+ if not isinstance(step, dict):
+ continue
+ if "parallel" in step:
+ yield from (s for s in step["parallel"] or [] if isinstance(s, dict))
+ else:
+ yield step
+
+
+def uses_action(step, action):
+ return str(step.get("uses", "")).startswith(action)
+
+
+def job_needs(config):
+ needs = config.get("needs") or []
+ return [needs] if isinstance(needs, str) else list(needs)
+
+
+def check_skip_conditions(workflow, required_checks):
+ """Return the problems with rust.yml's skip conditions and artifact steps."""
+ errors = []
+ jobs = {
+ key: config
+ for key, config in (workflow.get("jobs") or {}).items()
+ if isinstance(config, dict)
+ }
+ listed = frozenset().union(*(members for _, members, _ in SKIP_POLICIES))
+
+ for job in sorted(listed - set(jobs)):
+ errors.append(
+ f" - job `{job}` is listed in a skip set but does not exist "
+ f"in {RUST_WORKFLOW}"
+ )
+ for job, config in sorted(jobs.items()):
+ expected = expected_condition(job)
+ found = config.get("if")
+ if found == expected:
+ continue
+ if expected is None:
+ errors.append(
+ f" - job `{job}` is in no skip set, so it must carry no "
+ f"job-level `if:`; found {found!r}. List the job in a skip "
+ f"set or extend this check."
+ )
+ else:
+ errors.append(
+ f" - job `{job}` must carry exactly `if: {expected}`, "
+ f"found {found!r}"
+ )
+
+ for set_name, members, _ in SKIP_POLICIES:
+ for job, config in sorted(jobs.items()):
+ if job in members:
+ continue
+ blocked = sorted(set(job_needs(config)) & members)
+ if blocked:
+ errors.append(
+ f" - job `{job}` is not in {set_name} but needs "
+ f"{blocked}; GitHub would skip it with them"
+ )
+ # Only the merge queue gates a job that skips after the merge. A job
+ # that skips on a fork PR still runs on the fork, in the queue, and on
+ # main.
+ if required_checks and set_name == "POST_MERGE_SKIPPED_JOBS":
+ for job in sorted(members & set(jobs)):
+ name = jobs[job].get("name", job)
+ if name not in required_checks:
+ errors.append(
+ f" - job `{job}` ({name!r}) is in {set_name} but is "
+ f"not a required status check in .asf.yaml, so no "
+ f"run would gate it before it lands on main"
+ )
+
+ build = jobs.get(ARTIFACT_JOB)
+ if build is None:
+ errors.append(f" - job `{ARTIFACT_JOB}` does not exist in {RUST_WORKFLOW}")
+ else:
+ steps = list(iter_steps(build))
+ archive = [s for s in steps if s.get("id") == ARTIFACT_ARCHIVE_STEP]
+ uploads = [
+ s
+ for s in steps
+ if uses_action(s, "actions/upload-artifact")
+ and (s.get("with") or {}).get("name") == ARTIFACT_NAME
+ ]
+ if len(archive) != 1 or archive[0].get("if") != ARTIFACT_ARCHIVE_CONDITION:
+ found = [s.get("if") for s in archive]
+ errors.append(
+ f" - the step `id: {ARTIFACT_ARCHIVE_STEP}` in `{ARTIFACT_JOB}` "
+ f"must exist once and carry exactly "
+ f"`if: {ARTIFACT_ARCHIVE_CONDITION}`, found {found!r}"
+ )
+ if len(uploads) != 1 or uploads[0].get("if") != ARTIFACT_UPLOAD_CONDITION:
+ found = [s.get("if") for s in uploads]
+ errors.append(
+ f" - the `{ARTIFACT_NAME}` upload step in `{ARTIFACT_JOB}` must "
+ f"exist once and carry exactly `if: {ARTIFACT_UPLOAD_CONDITION}`, "
+ f"found {found!r}"
+ )
+
+ # The artifact is missing on both events, so every consumer must skip on both.
+ consumer_sets = [
+ (name, members)
+ for name, members, _ in SKIP_POLICIES
+ if name == "POST_MERGE_SKIPPED_JOBS" or ARTIFACT_JOB in members
+ ]
+ for job, config in sorted(jobs.items()):
+ downloads = [
+ s
+ for s in iter_steps(config)
+ if uses_action(s, "actions/download-artifact")
+ and (s.get("with") or {}).get("name") == ARTIFACT_NAME
+ ]
+ if not downloads:
+ continue
+ for set_name, members in consumer_sets:
+ if job not in members:
+ errors.append(
+ f" - job `{job}` downloads the `{ARTIFACT_NAME}` artifact "
+ f"but is not in {set_name}, where its producer skips"
+ )
+ return errors
+
+
+def main():
+ repo_root = os.path.dirname(
+ os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+ )
+ with open(os.path.join(repo_root, ".github", "workflows", RUST_WORKFLOW)) as f:
+ workflow = yaml.safe_load(f)
+ required_checks = get_required_checks(os.path.join(repo_root, ".asf.yaml"))
+
+ errors = check_skip_conditions(workflow, required_checks)
+ if errors:
+ print(f"ERROR: Problems found with skip conditions in {RUST_WORKFLOW}:\n")
+ print("\n".join(errors))
+ print()
+ sys.exit(1)
+
+ print(
+ f"OK: {RUST_WORKFLOW} skips exactly {len(POST_MERGE_SKIPPED_JOBS)} jobs "
+ f"on pushes to main and {len(FORK_PR_SKIPPED_JOBS)} jobs on pull "
+ "requests from forks."
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/ci/scripts/test_fork_ci_status.cjs b/ci/scripts/test_fork_ci_status.cjs
new file mode 100755
index 0000000000..46e55c1776
--- /dev/null
+++ b/ci/scripts/test_fork_ci_status.cjs
@@ -0,0 +1,373 @@
+#!/usr/bin/env node
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Test fork CI workflows with stubbed GitHub APIs. No API calls are made.
+const assert = require("node:assert/strict");
+const { execFileSync } = require("node:child_process");
+const { readFileSync } = require("node:fs");
+const path = require("node:path");
+const { test } = require("node:test");
+
+function loadWorkflow(name) {
+ return JSON.parse(
+ execFileSync(
+ "python3",
+ [
+ "-c",
+ "import json, sys, yaml; print(json.dumps(yaml.safe_load(sys.stdin)))",
+ ],
+ {
+ input: readFileSync(
+ path.join(__dirname, "../../.github/workflows", name)
+ ),
+ encoding: "utf8",
+ }
+ )
+ );
+}
+const script =
+ loadWorkflow("fork_ci_status.yml").jobs.bridge.steps[0].with.script;
+const rustWorkflow = loadWorkflow("rust.yml");
+const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor;
+const bridge = new AsyncFunction("github", "context", "core", "Date", script);
+const now = Date.UTC(2026, 0, 1);
+class Clock extends Date {
+ static now() {
+ return now;
+ }
+}
+const sha = "a".repeat(40);
+const pr = {
+ number: 1,
+ updated_at: new Date(now).toISOString(),
+ head: {
+ sha,
+ repo: { owner: { login: "contributor" }, name: "datafusion", fork: true },
+ },
+ base: { ref: "main" },
+};
+const run = (id, event = "push", conclusion = "success", run_attempt = 1) => ({
+ id,
+ event,
+ conclusion,
+ run_attempt,
+ status: "completed",
+ html_url: `https://github.com/contributor/datafusion/actions/runs/${id}`,
+});
+const checkFor = (run) => ({
+ id: 10,
+ status: run.status,
+ conclusion: run.conclusion,
+ output: {
+ text: JSON.stringify({
+ owner: "contributor",
+ repo: "datafusion",
+ head_sha: sha,
+ run_id: run.id,
+ run_attempt: run.run_attempt,
+ }),
+ },
+});
+
+async function execute({
+ check,
+ runs = [],
+ recordedRun,
+ event = "schedule",
+ workflowState = "active",
+ recentPrs = [pr],
+ slice = [pr],
+ calls = [],
+}) {
+ const writes = [];
+ const github = {
+ rest: {
+ checks: {
+ listForRef: async () => {
+ calls.push("checks.listForRef");
+ return { data: { check_runs: check ? [check] : [] } };
+ },
+ create: async (payload) =>
+ writes.push({ method: "create", ...payload }),
+ update: async (payload) =>
+ writes.push({ method: "update", ...payload }),
+ },
+ actions: {
+ listWorkflowRuns: async (params) => {
+ calls.push("actions.listWorkflowRuns");
+ assert.equal(params.owner, "contributor");
+ assert.equal(params.repo, "datafusion");
+ assert.equal(params.workflow_id, "rust.yml");
+ assert.equal(params.head_sha, sha);
+ return { data: { workflow_runs: runs } };
+ },
+ getWorkflowRun: async () => {
+ if (!recordedRun) throw new Error("Not Found");
+ return { data: recordedRun };
+ },
+ getWorkflow: async () => ({ data: { state: workflowState } }),
+ },
+ pulls: {
+ list: async ({ per_page }) => {
+ if (per_page === 1) return { data: [pr], headers: {} };
+ assert.equal(per_page, 25);
+ return { data: slice };
+ },
+ },
+ },
+ paginate: {
+ iterator: async function* () {
+ yield { data: recentPrs };
+ },
+ },
+ };
+ await bridge(
+ github,
+ {
+ repo: { owner: "apache", repo: "datafusion" },
+ eventName: event,
+ payload: { pull_request: pr },
+ },
+ { info: () => {} },
+ Clock
+ );
+ return writes;
+}
+
+function assertRun(write, method, expected) {
+ assert.equal(write.method, method);
+ assert.equal(write.status, expected.status);
+ assert.equal(write.conclusion, expected.conclusion);
+ assert.equal(write.details_url, expected.html_url);
+ assert.equal(JSON.parse(write.output.text).run_id, expected.id);
+ assert.equal(JSON.parse(write.output.text).run_attempt, expected.run_attempt);
+}
+
+test("notification selects a newer manual run instead of a push run", async () => {
+ const latest = run(2, "workflow_dispatch");
+ const writes = await execute({
+ event: "pull_request_target",
+ runs: [latest, run(1)],
+ });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "create", latest);
+});
+
+for (const conclusion of ["cancelled", "failure"]) {
+ test(`sweep replaces a ${conclusion} push run with a newer manual run`, async () => {
+ const previous = run(1, "push", conclusion);
+ const latest = run(2, "workflow_dispatch");
+ const writes = await execute({
+ check: checkFor(previous),
+ recordedRun: previous,
+ runs: [latest, previous],
+ });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "update", latest);
+ });
+}
+
+test("sweep recovers when the recorded run was deleted", async () => {
+ const latest = run(2, "workflow_dispatch");
+ const writes = await execute({ check: checkFor(run(1)), runs: [latest] });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "update", latest);
+});
+
+test("sweep creates a missing check from the PR head", async () => {
+ const latest = run(2);
+ const writes = await execute({ runs: [latest] });
+ assert.equal(writes.length, 1);
+ assert.equal(writes[0].head_sha, sha);
+ assertRun(writes[0], "create", latest);
+});
+
+for (const [state, conclusion] of [
+ ["active", "skipped"],
+ ["disabled_fork", "action_required"],
+]) {
+ test(`sweep creates a missing check when the workflow is ${state} and has no run`, async () => {
+ const writes = await execute({ workflowState: state });
+ assert.equal(writes.length, 1);
+ assert.equal(writes[0].method, "create");
+ assert.equal(writes[0].head_sha, sha);
+ assert.equal(writes[0].conclusion, conclusion);
+ assert.equal(JSON.parse(writes[0].output.text).run_id, null);
+ });
+}
+
+test("sweep picks up a rerun of the same workflow run", async () => {
+ const previous = run(1, "push", "failure");
+ const latest = run(1, "push", "success", 2);
+ const writes = await execute({
+ check: checkFor(previous),
+ recordedRun: latest,
+ runs: [latest],
+ });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "update", latest);
+});
+
+test("sweep does not rewrite an unchanged check", async () => {
+ const latest = run(1);
+ assert.deepEqual(
+ await execute({
+ check: checkFor(latest),
+ recordedRun: latest,
+ runs: [latest],
+ }),
+ []
+ );
+});
+
+for (const conclusion of ["success", "failure", "cancelled"]) {
+ test(`recent completed ${conclusion} checks wait for their sweep slice`, async () => {
+ const calls = [];
+ const previous = run(1, "push", conclusion);
+ const writes = await execute({
+ check: checkFor(previous),
+ runs: [run(2, "workflow_dispatch")],
+ slice: [],
+ calls,
+ });
+ assert.deepEqual(writes, []);
+ assert.deepEqual(calls, ["checks.listForRef"]);
+ });
+}
+
+test("the sweep finds a rerun on a recently updated completed PR", async () => {
+ const calls = [];
+ const latest = run(2, "workflow_dispatch");
+ const writes = await execute({
+ check: checkFor(run(1)),
+ runs: [latest],
+ calls,
+ });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "update", latest);
+ assert.deepEqual(calls, ["checks.listForRef", "actions.listWorkflowRuns"]);
+});
+
+test("the sweep still checks completed runs on older PRs", async () => {
+ const oldPr = {
+ ...pr,
+ updated_at: new Date(now - 24 * 3600 * 1000).toISOString(),
+ };
+ const latest = run(2, "workflow_dispatch");
+ const writes = await execute({
+ check: checkFor(run(1)),
+ runs: [latest],
+ recentPrs: [oldPr],
+ slice: [oldPr],
+ });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "update", latest);
+});
+
+for (const status of ["queued", "in_progress"]) {
+ test(`recent ${status} runs are polled outside the sweep slice`, async () => {
+ const previous = { ...run(1), status, conclusion: null };
+ const latest = run(1);
+ const writes = await execute({
+ check: checkFor(previous),
+ runs: [latest],
+ slice: [],
+ });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "update", latest);
+ });
+}
+
+test("recent checks with no fork run keep looking outside the sweep slice", async () => {
+ const missing = checkFor(run(null, "push", "skipped"));
+ const latest = run(2);
+ const writes = await execute({ check: missing, runs: [latest], slice: [] });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "update", latest);
+});
+
+test("recent missing checks are recovered outside the sweep slice", async () => {
+ const latest = run(2);
+ const writes = await execute({ runs: [latest], slice: [] });
+ assert.equal(writes.length, 1);
+ assertRun(writes[0], "create", latest);
+});
+
+test("a recent active PR in the sweep slice is only polled once", async () => {
+ const calls = [];
+ const active = { ...run(1), status: "in_progress", conclusion: null };
+ await execute({ check: checkFor(active), runs: [active], calls });
+ assert.deepEqual(calls, ["checks.listForRef", "actions.listWorkflowRuns"]);
+});
+
+// The concurrency expressions use string lookups and &&/||, whose semantics
+// match JavaScript for these non-empty strings and booleans.
+function concurrencyGroup(github) {
+ return rustWorkflow.concurrency.group.replace(
+ /\$\{\{\s*(.*?)\s*\}\}/g,
+ (_, expression) => new Function("github", `return (${expression});`)(github)
+ );
+}
+const forkPush = {
+ repository: "contributor/datafusion",
+ head_ref: "",
+ sha: "first",
+ ref: "refs/heads/feature",
+ workflow: "Rust",
+};
+
+test("successive pushes to a fork branch cancel the previous run", () => {
+ assert.equal(rustWorkflow.concurrency["cancel-in-progress"], true);
+ assert.equal(
+ concurrencyGroup(forkPush),
+ concurrencyGroup({ ...forkPush, sha: "second" })
+ );
+});
+
+test("different fork branches have separate concurrency groups", () => {
+ assert.notEqual(
+ concurrencyGroup(forkPush),
+ concurrencyGroup({ ...forkPush, ref: "refs/heads/other" })
+ );
+});
+
+for (const ref of [
+ "refs/heads/main",
+ "refs/heads/gh-readonly-queue/main/pr-1",
+]) {
+ test(`upstream ${ref} runs keep separate groups for each SHA`, () => {
+ const upstream = { ...forkPush, repository: "apache/datafusion", ref };
+ assert.notEqual(
+ concurrencyGroup(upstream),
+ concurrencyGroup({ ...upstream, sha: "second" })
+ );
+ });
+}
+
+test("upstream PR updates still share a concurrency group", () => {
+ const upstream = {
+ ...forkPush,
+ repository: "apache/datafusion",
+ head_ref: "feature",
+ ref: "refs/pull/1/merge",
+ };
+ assert.equal(
+ concurrencyGroup(upstream),
+ concurrencyGroup({ ...upstream, sha: "second" })
+ );
+});
diff --git a/ci/scripts/test_rust_workflow_skips.py b/ci/scripts/test_rust_workflow_skips.py
new file mode 100755
index 0000000000..2170439d98
--- /dev/null
+++ b/ci/scripts/test_rust_workflow_skips.py
@@ -0,0 +1,71 @@
+#!/usr/bin/env python3
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Regression tests for the Rust workflow skip policies."""
+
+import unittest
+from pathlib import Path
+
+import yaml
+
+from check_asf_yaml_status_checks import get_required_checks
+from check_rust_workflow_skips import check_skip_conditions
+
+ROOT = Path(__file__).resolve().parents[2]
+
+
+class RustWorkflowSkipsTest(unittest.TestCase):
+ def setUp(self):
+ self.workflow = yaml.safe_load((ROOT / ".github/workflows/rust.yml").read_text())
+ self.required_checks = get_required_checks(ROOT / ".asf.yaml")
+
+ def errors(self):
+ return check_skip_conditions(self.workflow, self.required_checks)
+
+ def test_current_workflow(self):
+ self.assertEqual(self.errors(), [])
+
+ def test_missing_skip_guard(self):
+ del self.workflow["jobs"]["linux-test"]["if"]
+ self.assertTrue(
+ any("job `linux-test` must carry exactly" in error for error in self.errors())
+ )
+
+ def test_unskipped_job_needs_skipped_job(self):
+ self.workflow["jobs"]["check-fmt"]["needs"] = "linux-test"
+ self.assertIn(
+ " - job `check-fmt` is not in FORK_PR_SKIPPED_JOBS but needs "
+ "['linux-test']; GitHub would skip it with them",
+ self.errors(),
+ )
+
+ def test_missing_artifact_upload_guard(self):
+ upload = next(
+ step
+ for step in self.workflow["jobs"]["linux-build-lib"]["steps"]
+ if step.get("uses", "").startswith("actions/upload-artifact")
+ and step.get("with", {}).get("name") == "cargo-check"
+ )
+ del upload["if"]
+ self.assertTrue(
+ any("`cargo-check` upload step" in error for error in self.errors())
+ )
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/dev/rust_lint.sh b/dev/rust_lint.sh
index dbb6454302..b1dddaac3b 100755
--- a/dev/rust_lint.sh
+++ b/dev/rust_lint.sh
@@ -88,10 +88,10 @@ done
SCRIPT_NAME="$(basename "${BASH_SOURCE[0]}")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-# `ci/scripts/check_asf_yaml_status_checks.py` runs with `python3` from PATH
-# and imports PyYAML. Report a missing prerequisite before any tool is
-# installed or any formatter runs, and point at `uv run`, which sets up the
-# Python dependencies from the uv workspace.
+# `ci/scripts/check_asf_yaml_status_checks.py` and `check_rust_workflow_skips.py`
+# run with `python3` from PATH and import PyYAML. Report a missing prerequisite
+# before any tool is installed or any formatter runs, and point at `uv run`,
+# which sets up the Python dependencies from the uv workspace.
ensure_python_with_yaml() {
if ! command -v python3 &> /dev/null; then
echo "[${SCRIPT_NAME}] python3 was not found on PATH. Please run the suite through uv, which provides Python and its packages: uv run ./dev/rust_lint.sh" >&2
@@ -136,6 +136,9 @@ declare -a WRITE_STEPS=(
declare -a READONLY_STEPS=(
"ci/scripts/check_no_cargo_install_in_workflows.sh|false"
"ci/scripts/check_asf_yaml_status_checks.py|false"
+ "ci/scripts/check_rust_workflow_skips.py|false"
+ "ci/scripts/test_rust_workflow_skips.py|false"
+ "ci/scripts/test_fork_ci_status.cjs|false"
"ci/scripts/check_large_files.sh|false"
"ci/scripts/markdown_link_check.sh|false"
"ci/scripts/security_audit.sh|false"
diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md
index 81c432013a..2efd13921d 100644
--- a/docs/source/contributor-guide/index.md
+++ b/docs/source/contributor-guide/index.md
@@ -239,3 +239,26 @@ According to Runs-On, spot instance termination is extremely rare for instances
### GitHub Runners
We also use standard GitHub runners for some actions in the main repository; these are also runnable in forks.
+
+#### Pull Requests from Forks
+
+DataFusion and its subprojects use substantial CI runner time. ASF Infrastructure
+identified them among its largest GitHub Actions users in
+[#25148](https://github.com/apache/datafusion/issues/25148). Fork CI reduces
+duplicate work on ASF's shared runners.
+
+A pull request from a fork to `main` runs the Rust workflow on your fork, with
+your own GitHub Actions runners. The main repository runs only the fast lint jobs for
+such a pull request and adds a `Rust (fork)` check that links to the run on
+your fork. The merge queue runs the full workflow on the merged result before
+a pull request merges, so a green fork run does not replace it.
+
+To make this work:
+
+1. Before you push your branch, open the **Actions** tab on your fork.
+ Confirm the prompt to enable workflows.
+2. Keep the `main` branch of your fork in sync with upstream. Runs on your
+ branches restore the build caches that a run on your `main` saves, so a
+ synced fork builds much faster.
+3. Rebase your branch before you push. The fork run tests your branch as it
+ is, while the merge queue tests it merged with `main`.
diff --git a/docs/source/contributor-guide/testing.md b/docs/source/contributor-guide/testing.md
index 3e39ef2eb1..f4b5c1cbc0 100644
--- a/docs/source/contributor-guide/testing.md
+++ b/docs/source/contributor-guide/testing.md
@@ -163,6 +163,22 @@ For local SQLite test instructions, see the [instructions in the documentation].
[instructions in the documentation]: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest#running-tests-sqlite
[extended.yml]: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml
+## CI Workflow Tests
+
+The Rust workflow validator and fork status bridge have regression tests.
+These tests cover skip conditions, fork run recovery, polling frequency, and
+concurrency groups for fork pushes and merge queue runs.
+They run in the `asf-yaml-check` CI job and in `./dev/rust_lint.sh`.
+Run them locally with Python, PyYAML, and Node.js 24:
+
+```shell
+python3 ci/scripts/test_rust_workflow_skips.py
+node --test ci/scripts/test_fork_ci_status.cjs
+```
+
+The validator tests remove guards and add an invalid job dependency in memory.
+The bridge tests use stubbed GitHub API responses and make no network requests.
+
## Rust Integration Tests
There are several public interface tests for the DataFusion library in the [tests](https://github.com/apache/datafusion/tree/main/datafusion/core/tests) directory.
@@ -261,18 +277,21 @@ bash ci/scripts/rust_docs.sh
## ASF Status Check Validation
`ci/scripts/check_asf_yaml_status_checks.py` checks that every required status
-check in `.asf.yaml` matches a job in `.github/workflows`, and that `rust.yml`
-skips only its listed jobs on pushes to `main`. `./dev/rust_lint.sh` runs it
-and needs `python3` with [PyYAML]. The [uv] workspace provides both:
+check in `.asf.yaml` matches a job in `.github/workflows`.
+`ci/scripts/check_rust_workflow_skips.py` checks that `rust.yml` skips only its
+listed jobs on pushes to `main` and on pull requests from forks.
+`./dev/rust_lint.sh` runs both and needs `python3` with [PyYAML]. The [uv]
+workspace provides both:
```shell
uv run ./dev/rust_lint.sh
```
-To run the check on its own:
+To run the checks on their own:
```shell
uv run python3 ci/scripts/check_asf_yaml_status_checks.py
+uv run python3 ci/scripts/check_rust_workflow_skips.py
```
[pyyaml]: https://pypi.org/project/PyYAML/
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org