You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/07/29 19:32:39 UTC
(texera) branch gh-readonly-queue/main/pr-7042-2b8df1277f32d108ba1ddf9458d3a43307c8ded7 created (now 706f831211)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-7042-2b8df1277f32d108ba1ddf9458d3a43307c8ded7
in repository https://gitbox.apache.org/repos/asf/texera.git
at 706f831211 ci: speed up merge queue via HEADGREEN and stack selection (#7042)
This branch includes the following new commits:
new 706f831211 ci: speed up merge queue via HEADGREEN and stack selection (#7042)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(texera) 01/01: ci: speed up merge queue via HEADGREEN and stack selection (#7042)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-7042-2b8df1277f32d108ba1ddf9458d3a43307c8ded7
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 706f831211259929fe4c418ac813b1eda24e7685
Author: Yicong Huang <17...@users.noreply.github.com>
AuthorDate: Wed Jul 29 15:32:18 2026 -0400
ci: speed up merge queue via HEADGREEN and stack selection (#7042)
### What changes were proposed in this PR?
Two complementary merge-queue speedups.
**1. Retune the `merge_queue` ruleset in `.asf.yaml`:**
| Parameter | Before | After | Why |
| --- | --- | --- | --- |
| `grouping_strategy` | ALLGREEN | HEADGREEN | Merge a whole batch on
one green head-group run instead of one green run per queued PR; also
cuts flaky-test exposure from N draws per batch to 1. |
| `max_entries_to_build` | 5 | 2 | Under HEADGREEN, non-head speculative
builds no longer gate merging; keep one in flight for pipelining, save
shared ASF runner quota (the pool has been at its 900-job limit lately;
policy caps a project at 20 concurrent jobs, and one full group is
already ~20). |
| `min_entries_to_merge` | 1 | 2 | Batch PRs so one full CI run merges
several at peak times. |
| `min_entries_to_merge_wait_minutes` | 5 | 3 | Short batching window;
off-peak a lone PR waits at most 3 minutes. |
| `check_response_timeout_minutes` | 30 | 45 | Headroom for runner-queue
delays so a late-reporting check is not misjudged as failed. |
**2. Smarter stack selection for `merge_group` runs (precheck.yml).**
Previously merge groups ran every stack unconditionally. Now, per group:
```
merge_group
├─ resolve the group's PRs from squash messages "(#N)" (same convention as direct-backport-push.yml)
├─ tier 1: group tree == a PR head tree that already passed Required Checks
│ → byte-identical content was just validated → skip every stack
├─ tier 2: otherwise, union the labels of every PR in the group and apply
│ the same LABEL_STACKS filter a PR event gets
└─ any resolution failure → fail open, run every stack
```
Tier 1 typically fires for a single fresh PR based on the current `main`
tip (or a stacked PR containing the group's other entries) — tree
equality makes the skip content-exact, not heuristic. Backport targets
stay PR-event-only; merge-group backports are still discovered
post-merge by `direct-backport-push.yml`, which already iterates every
commit in a multi-PR push.
### Any related issues, documentation, discussions?
Closes #7041. Ruleset parameters follow the GitHub rulesets
`merge_queue` schema: https://docs.github.com/en/rest/repos/rules
### How was this PR tested?
- Config-only + workflow-logic change; `.asf.yaml` and both workflows
pass YAML parsing, and the embedded github-script block passes `node
--check`.
- The merge-group path fails open on any API/parse error (single
try/catch → full run), so a bad lookup can never skip CI silently.
- Real merge_group behavior can only be observed on `main`; will watch
the first few queue batches after merge (skip/label-union decisions are
logged in the precheck step).
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)
---------
Co-authored-by: Claude Fable 5 <no...@anthropic.com>
---
.asf.yaml | 10 +--
.github/workflows/precheck.yml | 125 ++++++++++++++++++++++++++++++++--
.github/workflows/required-checks.yml | 6 +-
3 files changed, 129 insertions(+), 12 deletions(-)
diff --git a/.asf.yaml b/.asf.yaml
index 8a23cb6f45..c0f520b666 100644
--- a/.asf.yaml
+++ b/.asf.yaml
@@ -83,12 +83,12 @@ github:
- type: merge_queue
parameters:
merge_method: SQUASH
- max_entries_to_build: 5
- min_entries_to_merge: 1
+ max_entries_to_build: 2
+ min_entries_to_merge: 2
max_entries_to_merge: 5
- min_entries_to_merge_wait_minutes: 5
- grouping_strategy: ALLGREEN
- check_response_timeout_minutes: 30
+ min_entries_to_merge_wait_minutes: 3
+ grouping_strategy: HEADGREEN
+ check_response_timeout_minutes: 45
- type: pull_request
parameters:
allowed_merge_methods:
diff --git a/.github/workflows/precheck.yml b/.github/workflows/precheck.yml
index c98589060b..4ce1d24092 100644
--- a/.github/workflows/precheck.yml
+++ b/.github/workflows/precheck.yml
@@ -19,9 +19,12 @@
# Checks (backport preflight/build) call, so the label -> stack mapping and
# the labeler-wait live in exactly one place. On PR events it waits for the
# Pull Request Labeler to finish, then maps the PR's labels to the build
-# stacks each requires and to the release/* backport targets. Non-PR events
-# (push / merge_group / workflow_dispatch) run every stack and request no
-# backports.
+# stacks each requires and to the release/* backport targets. Push and
+# workflow_dispatch events run every stack. Merge groups resolve the PRs in
+# the group and either skip every stack (group tree identical to a PR head
+# that already passed Required Checks) or apply the same label -> stack
+# filter using the union of the group's PR labels; resolution failures fall
+# open to a full run. Non-PR events request no backports.
name: Precheck
on:
@@ -50,6 +53,9 @@ on:
permissions:
checks: read
+ # contents: read backs the git-data commit lookups (tree SHAs) in the
+ # merge-queue fast path.
+ contents: read
pull-requests: read
jobs:
@@ -100,6 +106,97 @@ jobs:
const eventName = context.eventName;
let labels = [];
+ // When true, the LABEL_STACKS filter below applies to `labels`;
+ // when false the event runs every stack.
+ let applyLabelFilter = false;
+
+ // Merge-queue path, two tiers, evaluated per group:
+ //
+ // 1. Fast path — a merge group whose squash result has the same
+ // git tree as a PR head that already passed Required Checks
+ // would re-validate byte-identical content: skip every stack.
+ // Tree equality only holds for a single-entry group whose PR
+ // was based on the then-current main (or a stacked PR that
+ // contains the group's other entries), so the skip is
+ // content-exact rather than heuristic.
+ // 2. Label union — otherwise, resolve every PR in the group from
+ // its squash commit message (PR_TITLE_AND_DESC squash titles
+ // end with "(#N)", same convention direct-backport-push.yml
+ // relies on) and apply the same label -> stack filter a PR
+ // event gets, using the union of all the group's PR labels.
+ //
+ // Any resolution failure falls open to a full run.
+ let mergeGroupAlreadyValidated = false;
+ if (eventName === "merge_group") {
+ try {
+ const { owner, repo } = context.repo;
+ const mg = context.payload.merge_group;
+ const { data: comparison } = await github.rest.repos.compareCommits({
+ owner,
+ repo,
+ base: mg.base_sha,
+ head: mg.head_sha,
+ });
+ const prNumbers = comparison.commits.map((c) => {
+ const m = c.commit.message.split("\n", 1)[0].match(/\(#(\d+)\)\s*$/);
+ if (!m) throw new Error(`no PR number in the message of ${c.sha}`);
+ return Number(m[1]);
+ });
+ if (prNumbers.length === 0) {
+ throw new Error(`no commits between ${mg.base_sha} and ${mg.head_sha}`);
+ }
+ const pullRequests = [];
+ for (const pullNumber of prNumbers) {
+ const { data: pr } = await github.rest.pulls.get({
+ owner,
+ repo,
+ pull_number: pullNumber,
+ });
+ pullRequests.push(pr);
+ }
+
+ // Tier 1: tree match against the newest entry's PR head.
+ const headPr = pullRequests[pullRequests.length - 1];
+ const mgTree = (
+ await github.rest.git.getCommit({ owner, repo, commit_sha: mg.head_sha })
+ ).data.tree.sha;
+ const prTree = (
+ await github.rest.git.getCommit({ owner, repo, commit_sha: headPr.head.sha })
+ ).data.tree.sha;
+ if (mgTree === prTree) {
+ // The aggregator job in required-checks.yml publishes the
+ // "Required Checks" check run on the PR head.
+ const { data: checkData } = await github.rest.checks.listForRef({
+ owner,
+ repo,
+ ref: headPr.head.sha,
+ check_name: "Required Checks",
+ });
+ const check = checkData.check_runs[0];
+ if (check && check.status === "completed" && check.conclusion === "success") {
+ mergeGroupAlreadyValidated = true;
+ core.info(
+ `Merge group tree matches PR #${headPr.number} head ${headPr.head.sha}, which already passed Required Checks; skipping all stacks.`
+ );
+ }
+ }
+
+ // Tier 2: label union across the group's PRs.
+ if (!mergeGroupAlreadyValidated) {
+ labels = [...new Set(pullRequests.flatMap((pr) => pr.labels.map((l) => l.name)))];
+ applyLabelFilter = true;
+ core.info(
+ `Merge group PRs: ${prNumbers.map((n) => `#${n}`).join(", ")}; label union: ${labels.join(", ") || "(none)"}`
+ );
+ }
+ } catch (e) {
+ core.warning(`Merge-group PR resolution failed (${e.message}); running all stacks.`);
+ labels = [];
+ applyLabelFilter = false;
+ mergeGroupAlreadyValidated = false;
+ }
+ }
+
if (eventName === "pull_request") {
// Re-fetch labels: the labeler may have just added some.
const { data: pr } = await github.rest.pulls.get({
@@ -108,6 +205,7 @@ jobs:
pull_number: context.payload.pull_request.number,
});
labels = pr.labels.map((l) => l.name);
+ applyLabelFilter = true;
core.info(`PR labels: ${labels.join(", ") || "(none)"}`);
}
@@ -183,7 +281,7 @@ jobs:
let runInfra = true;
let runPyrightLanguageService = true;
- if (eventName === "pull_request") {
+ if (applyLabelFilter) {
const stacks = new Set();
for (const label of labels) {
for (const stack of LABEL_STACKS[label] || []) {
@@ -204,6 +302,18 @@ jobs:
);
}
+ if (mergeGroupAlreadyValidated) {
+ runFrontend = false;
+ runAmber = false;
+ runAmberIntegration = false;
+ runPlatform = false;
+ runPlatformIntegration = false;
+ runPyamber = false;
+ runAgentService = false;
+ runInfra = false;
+ runPyrightLanguageService = false;
+ }
+
core.setOutput("run_frontend", runFrontend ? "true" : "false");
core.setOutput("run_amber", runAmber ? "true" : "false");
core.setOutput("run_amber_integration", runAmberIntegration ? "true" : "false");
@@ -219,7 +329,12 @@ jobs:
// a feature that does not exist on any release branch). The veto
// wins even if release/* labels are also present, so adding it
// mid-review cancels the backport apply-check/build on the re-run.
- let targets = [...new Set(labels.filter((n) => /^release\/.+$/.test(n)))].sort();
+ // PR events only: merge_group also populates `labels` now, but its
+ // backports are discovered post-merge by direct-backport-push.yml.
+ let targets =
+ eventName === "pull_request"
+ ? [...new Set(labels.filter((n) => /^release\/.+$/.test(n)))].sort()
+ : [];
if (labels.includes("no-backport-needed")) {
core.info("no-backport-needed present; skipping all backport targets.");
targets = [];
diff --git a/.github/workflows/required-checks.yml b/.github/workflows/required-checks.yml
index f2ee2efb89..ae47606443 100644
--- a/.github/workflows/required-checks.yml
+++ b/.github/workflows/required-checks.yml
@@ -43,8 +43,10 @@ concurrency:
cancel-in-progress: ${{ github.ref != 'refs/heads/main' && !startsWith(github.ref, 'refs/heads/release/') }}
jobs:
- # Decide which build stacks this event needs (label union on PR events; all
- # stacks on push/merge_group/dispatch). Shared with Backport Checks via the
+ # Decide which build stacks this event needs (label union on PR events and
+ # merge groups — a merge group unions the labels of every PR it contains,
+ # and skips every stack when its tree matches an already-green PR head;
+ # all stacks on push/dispatch). Shared with Backport Checks via the
# reusable precheck workflow so the label -> stack mapping lives in one place
# (.github/workflows/precheck.yml). The backport_targets output is unused
# here — backport preflight/build moved to the Backport Checks workflow.