You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/08/10 08:40:21 UTC
(spark) branch branch-4.3 updated: [SPARK-58660][INFRA] Self-heal the PR Build check when the notify workflow fails to create it
This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 93e5bf4b3d09 [SPARK-58660][INFRA] Self-heal the PR Build check when the notify workflow fails to create it
93e5bf4b3d09 is described below
commit 93e5bf4b3d09369bc7b4f0970d37465b8b636507
Author: Ganesha S <ga...@gmail.com>
AuthorDate: Mon Aug 10 16:39:30 2026 +0800
[SPARK-58660][INFRA] Self-heal the PR Build check when the notify workflow fails to create it
### What changes were proposed in this pull request?
The PR "Build" status check is created by `notify_test_workflow.yml`, which runs once per push. This PR makes the scheduled `update_build_status.yml` job self-healing: when a PR has no "Build" check on its head commit, it now recreates one instead of only updating existing checks.
It looks up the fork's `build_main.yml` runs filtered by the head SHA (re-polling briefly to ride out registration lag):
- a run exists -> backfill a `queued` "Build" check that a later pass syncs, mirroring `notify_test_workflow.yml`;
- no run (Actions disabled, old-master branch, or `build_main.yml` missing) -> create the same `action_required` check with contributor instructions that `notify_test_workflow.yml` creates.
Transient lookup failures (5xx, network, or 403/429 rate limits) are retried on a later scheduled pass rather than reported as a fork misconfiguration. Deleted head repos and existing "Build" checks are left untouched.
### Why are the changes needed?
`notify_test_workflow.yml` is the only place that *creates* the "Build" check, and it is a single one-shot `pull_request_target` job with no retry. When it fails to complete -- most commonly when it is starved of an ASF shared runner and cancelled before running any step -- the check is never created, the scheduled updater has nothing to update, and the PR is stuck at `mergeable_state: unstable` with no required status and no automatic recovery.
The only workaround today is for the author to manually re-fire a `pull_request_target` event (push a new commit, or close and reopen the PR).
Example: PR apache/spark#57708 (head `b3d29d1`). All fork test checks were green, but the apache-side "Notify test workflow" run sat ~45 minutes and was cancelled with zero steps executed, so no "Build" check was ever created and the PR stayed pending indefinitely.
### Does this PR introduce _any_ user-facing change?
No. This only affects the CI infrastructure workflow that maintains the PR "Build" status check.
### How was this patch tested?
This is a scheduled `pull_request_target` workflow and cannot run in PR CI, so it was validated locally:
- YAML parses and the embedded `actions/github-script` body passes `node --check`.
- The recreation decision logic was exercised with a standalone simulation covering every branch: run present, empty-then-run-appears (registration lag), Actions disabled / empty result, 404 workflow missing, permanent 403/451, rate-limit 403 (retry-after / `x-ratelimit-remaining: 0` / secondary-limit message), 429, 5xx, and network errors -- each resolving to backfill / action_required / retry as expected.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57867 from ganeshashree/SPARK-58660.
Lead-authored-by: Ganesha S <ga...@gmail.com>
Co-authored-by: Ganesha S <ga...@databricks.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit f805e772801a06a5a040025306fb8ca0ab50d4f1)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.github/workflows/update_build_status.yml | 314 +++++++++++++++++++++++++++---
1 file changed, 285 insertions(+), 29 deletions(-)
diff --git a/.github/workflows/update_build_status.yml b/.github/workflows/update_build_status.yml
index d2320c59cce9..1bb2775235b3 100644
--- a/.github/workflows/update_build_status.yml
+++ b/.github/workflows/update_build_status.yml
@@ -46,6 +46,72 @@ jobs:
// See https://docs.github.com/en/graphql/reference/enums#mergestatestatus
const maybeReady = ['behind', 'clean', 'draft', 'has_hooks', 'unknown', 'unstable'];
+ // A fork workflow-run lookup can fail transiently (server errors, network drops, or
+ // REST rate limiting, which GitHub returns as 403 with rate-limit headers or 429).
+ // These should be retried on a later scheduled pass rather than reported to the
+ // contributor as a broken fork. Any other failure (e.g. a 404 for a missing
+ // build_main.yml) is treated as permanent.
+ // https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
+ const isTransientError = (e) => {
+ if (!e.status) return true;
+ if (e.status >= 500 || e.status == 429) return true;
+ if (e.status == 403) {
+ const headers = (e.response && e.response.headers) || {};
+ return !!headers['retry-after'] || headers['x-ratelimit-remaining'] === '0'
+ || /rate limit/i.test(e.message || '');
+ }
+ return false;
+ };
+
+ // List all check-runs for a commit. per_page=100 (not the default 30) matches
+ // notify_test_workflow.yml: a SHA can accumulate more check-runs than one page
+ // (CI matrix, external checks, duplicate Build checks from reopened PRs), which
+ // could otherwise push the target Build check off the first page and leave the PR
+ // stuck in 'queued' forever.
+ const listCheckRuns = (ref) => github.paginate(
+ 'GET /repos/{owner}/{repo}/commits/{ref}/check-runs',
+ {
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ ref: ref,
+ per_page: 100
+ }
+ );
+
+ // Parse a Build check's output text into the {owner, repo, run_id} the run fetch
+ // needs, or return null if it is absent, malformed, or missing a field. JSON.parse
+ // succeeding is not enough: a check from an older version, a manual run, or another
+ // app can carry null, {}, or unrelated JSON that parses but lacks these fields, and
+ // the run fetch would then fail.
+ const parseRunParams = (cr) => {
+ let params;
+ try {
+ params = JSON.parse(cr.output.text);
+ } catch (error) {
+ return null;
+ }
+ if (!params || !params.owner || !params.repo || !params.run_id) {
+ return null;
+ }
+ return params;
+ };
+
+ // A Build check this updater can actually sync: not action_required (notify writes
+ // that when it missed the fork run, and it carries no run params) and with output
+ // text carrying the {owner, repo, run_id} needed to fetch the fork run. A Build check
+ // with empty/malformed/fieldless output can never be synced, so it must not count as
+ // present - otherwise it would suppress the backfill and leave the PR stuck with an
+ // unsyncable check.
+ const isSyncableBuildCheck = (cr) =>
+ cr.name == 'Build' && cr.conclusion != 'action_required'
+ && parseRunParams(cr) != null;
+
+ // An action_required Build check: the contributor-facing "enable Actions / rebase"
+ // status notify writes when it found no fork run. It is not syncable, but it already
+ // carries the guidance the no-run backfill branch would create, so it is "useful".
+ const isActionRequiredBuildCheck = (cr) =>
+ cr.name == 'Build' && cr.conclusion == 'action_required';
+
// Iterate open PRs
for await (const prs of github.paginate.iterator(endpoint,params)) {
// Each page
@@ -53,35 +119,38 @@ jobs:
console.log('SHA: ' + pr.head.sha)
console.log(' Mergeable status: ' + pr.mergeable_state)
if (pr.mergeable_state == null || maybeReady.includes(pr.mergeable_state)) {
- // Paginate with per_page=100 to match notify_test_workflow.yml. The default
- // page size is 30, and a SHA can accumulate more check-runs than that (CI
- // matrix, external checks, duplicate Build checks from reopened PRs), which
- // could push the target Build check off the first page and leave the PR
- // stuck in 'queued' forever.
- const checkRuns = await github.paginate(
- 'GET /repos/{owner}/{repo}/commits/{ref}/check-runs',
- {
- owner: context.repo.owner,
- repo: context.repo.repo,
- ref: pr.head.sha,
- per_page: 100
- }
- )
+ const checkRuns = await listCheckRuns(pr.head.sha)
- // Iterator GitHub Checks in the PR
+ // Does this SHA already carry an action_required Build check? notify (or an
+ // earlier pass of this updater) writes one when no fork run was found. It is
+ // not syncable, so it never suppresses the backfill below - which means a PR
+ // that permanently lacks a fork run (Actions disabled, old master) would
+ // otherwise re-poll on every 15-minute pass forever. When one is already
+ // present, the backfill's re-poll is unnecessary (see below).
+ const hasActionRequiredBuildCheck = checkRuns.some(isActionRequiredBuildCheck)
+
+ // Track whether a syncable Build check exists (see isSyncableBuildCheck).
+ // notify_test_workflow.yml creates one per push; if that job never completed
+ // (e.g. cancelled while starved of an ASF runner) the check is missing and the
+ // backfill after this loop recreates it. Sync every match (no early break): a
+ // SHA can carry more than one Build check (reopened PRs, or a backfill that
+ // raced notify). Branch protection evaluates the newest check-run of a given
+ // name, so syncing only the first would leave a newer duplicate stuck in
+ // 'queued' and block the PR.
+ let syncableBuildCheck = false
+ // Build checks whose referenced fork run is permanently gone (404). They parse
+ // as syncable (their output still carries valid run params) but can never be
+ // synced, so the recheck below must not let them suppress the backfill.
+ const staleCheckIds = new Set()
for await (const cr of checkRuns) {
if (cr.name == 'Build' && cr.conclusion != "action_required") {
- // text contains parameters to make request in JSON. A Build check
- // created by something other than notify_test_workflow.yml (an older
- // version, a manual run, or another app) may have empty or malformed
- // output text; skip it instead of aborting the whole scheduled run,
- // which would block updates for every PR queued behind it.
- let params
- try {
- params = JSON.parse(cr.output.text)
- } catch (error) {
- console.error('Skipping Build check ' + cr.id + ' with unparseable output text')
- console.error(error)
+ // Skip a check with unusable output (see parseRunParams) instead of
+ // aborting the whole scheduled run, which would block every PR queued
+ // behind it. Leaving syncableBuildCheck false lets the backfill below
+ // replace it rather than stranding the PR.
+ const params = parseRunParams(cr)
+ if (!params) {
+ console.error('Skipping Build check ' + cr.id + ' with unusable output')
continue
}
@@ -91,10 +160,22 @@ jobs:
run = await github.request('GET /repos/{owner}/{repo}/actions/runs/{run_id}', params)
} catch (error) {
console.error(error)
- // Run not found. This can happen when the PR author removes GitHub Actions runs or
- // disables GitHub Actions.
+ // The referenced fork run could not be fetched. A transient failure (5xx,
+ // rate limit) should be retried on a later pass, so count the check as
+ // syncable to suppress the backfill this pass. A permanent not-found (404:
+ // the PR author deleted the fork run or disabled GitHub Actions) means this
+ // check can never be synced, so leave syncableBuildCheck false and let the
+ // backfill below recreate a useful status - otherwise the flag would stay
+ // set and leave Build 'queued' indefinitely on every pass.
+ if (isTransientError(error)) {
+ syncableBuildCheck = true
+ } else {
+ staleCheckIds.add(cr.id)
+ }
continue
}
+ // Only now that the run is fetched is this a check we can actually sync.
+ syncableBuildCheck = true
// Keep syncing the status of the checks
if (run.data.status == 'completed') {
@@ -119,8 +200,183 @@ jobs:
details_url: run.data.details_url
})
}
+ }
+ }
- break
+ // No syncable Build check: notify_test_workflow.yml never created one, or only
+ // a stale action_required one exists. Recreate it here, mirroring notify. Skip
+ // if the head repo was deleted.
+ if (!syncableBuildCheck && pr.head.repo) {
+ const forkOwner = pr.head.repo.owner.login
+ const forkRepo = pr.head.repo.name
+ // Look up the fork's build_main.yml runs for the PR branch, matching
+ // notify_test_workflow.yml. Filter by branch (head.ref), not just head_sha:
+ // build_main.yml skips fork pushes to master (the "Sync fork" case), so the
+ // same SHA can carry a skipped/unrelated run on another branch, and a
+ // head_sha-only lookup could attach the Build check to that run instead of
+ // the PR branch's. Then pick the run whose head_sha equals the settled head
+ // SHA so stale runs from earlier commits on the branch cannot drive the
+ // decision. Re-poll a few times: the run is often not yet registered right
+ // after a push, and we must not create the sticky action_required check below
+ // over registration lag. A transient lookup error (5xx, network, rate limit)
+ // aborts to a later scheduled pass; any other error is treated as "no runs".
+ //
+ // When an action_required check already exists that lag risk is already
+ // realized, so a single sleepless lookup suffices - this stops a PR that
+ // permanently lacks a fork run (Actions disabled, old master) from re-polling
+ // (3 lookups plus two 3s sleeps) on every 15-minute pass forever, while still
+ // promoting it to a queued check on whichever pass first sees a run appear.
+ const attempts = hasActionRequiredBuildCheck ? 1 : 3
+ let matched_run
+ let transient = false
+ for (let attempt = 0; attempt < attempts; attempt++) {
+ let forkRuns
+ try {
+ const runs = await github.request(
+ 'GET /repos/{owner}/{repo}/actions/workflows/{id}/runs',
+ {
+ owner: forkOwner,
+ repo: forkRepo,
+ id: 'build_main.yml',
+ branch: pr.head.ref
+ }
+ )
+ forkRuns = runs.data.workflow_runs
+ } catch (error) {
+ console.error(error)
+ // Transient -> retry on a later pass. Permanent (e.g. a 404 for a
+ // missing build_main.yml) -> there is no run to find and it will not
+ // appear in 3s, so stop polling with matched_run undefined and let the
+ // action_required branch below handle it. Either way, do not burn the
+ // remaining retries and sleeps on an error that cannot resolve here.
+ if (isTransientError(error)) {
+ transient = true
+ }
+ break
+ }
+ matched_run = forkRuns.find(r => r.head_sha == pr.head.sha)
+ if (matched_run) {
+ break
+ }
+ if (attempt < attempts - 1) {
+ await new Promise(resolve => setTimeout(resolve, 3000))
+ }
+ }
+ if (transient) {
+ console.log(' Fork run lookup failed transiently; will retry next pass')
+ continue
+ }
+
+ // An action_required Build check already exists and we still found no run:
+ // the guidance check the no-run branch below would create is already present,
+ // so there is nothing to backfill. Skip the recheck listing and the create
+ // outright. (This is only reachable via the single, sleepless lookup above,
+ // so there was no ~6s window for notify to have created a syncable check in
+ // the meantime; even if it had, skipping is safe - with no run there is
+ // nothing to promote to 'queued' this pass anyway.)
+ if (hasActionRequiredBuildCheck && !matched_run) {
+ console.log(' action_required Build check present; nothing to backfill')
+ continue
+ }
+
+ // Recheck for an existing Build check immediately before creating one. The
+ // poll above can span up to ~6 seconds, during which notify_test_workflow.yml
+ // may create the check; this recheck skips the backfill in that common case.
+ // It narrows but cannot fully close the window (there is no atomic
+ // create-if-absent, and the list endpoint is eventually consistent) - the
+ // sync loop above stays tolerant of duplicates so any that slip through still
+ // converge.
+ //
+ // Skip the backfill only if a check that is already at least as useful as the
+ // one we would create exists. When we found a fork run, that means a syncable
+ // check (the queued check we would create). When we found no run, that means a
+ // syncable OR an action_required check (the guidance check we would create).
+ // In both cases a malformed/fieldless check does NOT count - the sync loop
+ // cannot sync it, so refusing here would strand the PR on an unsyncable check;
+ // we fall through and create a usable one instead. A check whose fork run the
+ // sync loop just found permanently gone (staleCheckIds) is likewise not useful,
+ // even though its output still parses as syncable, so it too is excluded.
+ //
+ // The recheck listing and the create below are wrapped so a failure on one
+ // PR degrades to "skip this PR, retry next pass" rather than aborting the
+ // whole scheduled run and starving every PR later in the iteration - the same
+ // reason the sync loop above guards its calls.
+ try {
+ const recheck = await listCheckRuns(pr.head.sha)
+ const existingBuild = recheck.some(cr =>
+ (isSyncableBuildCheck(cr) && !staleCheckIds.has(cr.id))
+ || (!matched_run && isActionRequiredBuildCheck(cr)))
+ if (existingBuild) {
+ console.log(' Build check appeared during polling; skipping backfill')
+ continue
+ }
+
+ if (matched_run) {
+ // A run exists for this head commit; point the check at it and let the
+ // next pass sync its status.
+ const actions_url = 'https://github.com/' + forkOwner + '/' + forkRepo
+ + '/actions/runs/' + matched_run.id
+ console.log(' Backfilling missing Build check -> ' + actions_url)
+ await github.rest.checks.create({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ name: 'Build',
+ head_sha: pr.head.sha,
+ status: 'queued',
+ output: {
+ title: 'Test results',
+ summary: '[See test results](' + actions_url + ')\n\n'
+ + 'If the tests fail for reasons unrelated to this pull request, '
+ + 'please rerun the workflow in your forked repository.\n'
+ + 'If the failures are related to this pull request, '
+ + 'please investigate them and push follow-up changes.',
+ text: JSON.stringify({
+ owner: forkOwner,
+ repo: forkRepo,
+ run_id: matched_run.id
+ })
+ },
+ details_url: actions_url
+ })
+ } else {
+ // No run for this head commit after re-polling (empty result, permanent
+ // lookup failure, or build_main.yml missing): Actions is disabled, the
+ // branch is on an old master, or the commit never triggered a run. Mirror
+ // notify's action_required check so the PR carries a required status
+ // telling the contributor how to fix it.
+ console.log(' No forked run for head SHA; creating action_required check')
+ await github.rest.checks.create({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ name: 'Build',
+ head_sha: pr.head.sha,
+ status: 'completed',
+ conclusion: 'action_required',
+ output: {
+ title: 'Workflow run detection failed',
+ summary: `
+ Unable to detect the workflow run for testing the changes in your PR.
+
+ 1. If you did not enable GitHub Actions in your forked repository, please enable it by clicking the button as shown in the image below. See also [Managing Github Actions Settings for a repository](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository) for more details.
+ 2. It is possible your branch is based on the old \`master\` branch in Apache Spark, please sync your branch to the latest master branch. For example as below:
+ \`\`\`bash
+ git fetch upstream
+ git rebase upstream/master
+ git push origin YOUR_BRANCH --force
+ \`\`\``,
+ images: [
+ {
+ alt: 'enabling workflows button',
+ image_url: 'https://raw.githubusercontent.com/apache/spark/master/.github/workflows/images/workflow-enable-button.png'
+ }
+ ]
+ }
+ })
+ }
+ } catch (error) {
+ console.error(' Backfill failed for this PR; will retry next pass')
+ console.error(error)
+ continue
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org