You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@maka.apache.org by as...@apache.org on 2026/08/27 03:04:11 UTC

(maka) branch main updated: feat(ci): label pull requests by reading effort (#3949)

This is an automated email from the ASF dual-hosted git repository.

Astro-Han pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/maka.git


The following commit(s) were added to refs/heads/main by this push:
     new 4b8bafca7 feat(ci): label pull requests by reading effort (#3949)
4b8bafca7 is described below

commit 4b8bafca7b878555960ab5b3231fc9ed2fbbbe9e
Author: AstroHan <le...@outlook.com>
AuthorDate: Thu Aug 27 11:04:07 2026 +0800

    feat(ci): label pull requests by reading effort (#3949)
    
    * feat(ci): label pull requests by review state and reading effort
    
    The pull request list carries no labels at all, so finding the work that
    actually needs a reviewer means opening pull requests one at a time. At the
    current rate that scan is repeated over more than a hundred open pull
    requests, and two facts it depends on are not visible from the list at all:
    which stalled runs are waiting on a maintainer rather than on the author, and
    which approvals have already been retired by a later push.
    
    Project each pull request onto two independent axes instead.
    
    Status answers whose move it is. `waiting-infra` covers a run held by the fork
    approval gate or never created, which the contributor cannot clear; the gate
    outranks a sibling failure because no amount of fixing produces a complete
    verdict while a run is held. `needs-author` covers a conclusive failure or a
    standing request for changes. `review-ready` is green and unapproved, and is
    the only queue a reviewer needs to read. `merge-ready` additionally carries a
    committer approval on the current head.
    
    Effort answers how much reading a pull request is, on tiers that map to the
    only two decisions a reviewer makes about size: whether it can be cleared
    between other work, and whether it is too large to read as one unit.
    Lockfiles, regenerated artifacts and binaries are excluded from the count,
    since a pull request that touches one would otherwise land several tiers above
    what it asks anyone to read. Test code is not discounted; it is reviewed too.
    
    Approvals and requests for changes are treated asymmetrically on purpose. An
    approval binds to the head it was given on, so a push retires it even where
    stale-review dismissal is off. A request for changes survives a push, or
    pushing anything at all would clear the block.
    
    The labels are a projection and nothing more. Branch protection remains the
    only merge authority, and no label here grants or withholds it. A draft is
    left unlabelled rather than having a check result overwrite the author's own
    statement that it is not ready.
    
    `scripts/pr-triage-dryrun.mjs` reports the same classification over live pull
    requests without writing anything, so a rule change is reviewed against real
    data before it reaches anyone's pull request.
    
    Generated-by: Claude Code
    
    * feat(ci): label pull requests by reading effort
    
    Reviewer attention is the binding constraint here, and nothing in the pull
    request list says how much reading each entry asks for. Sorting that out means
    opening pull requests one at a time, over a set that turns over roughly fifty
    times a day.
    
    Label the one axis a search cannot express. GitHub already indexes review,
    check and draft state, so `review:required status:success` answers "whose move
    is it" for free and stays correct without anything keeping a copy fresh. There
    is no size qualifier, so effort has to be computed and stored.
    
    Five tiers, on inclusive upper bounds of readable lines: XS 10, S 100, M 500,
    L 1000, XL above. Every tier is populated on the current open set (5/26/41/21/49),
    so none of them is decoration.
    
    Readable lines exclude what nobody reads line by line: lockfiles including
    Cargo.lock, both spellings the notice generator emits, regenerated sources,
    snapshots and binaries. Without that a Rust dependency bump reads as XL on the
    strength of 3524 rewritten lock lines. Test code is not discounted; it is
    reviewed too.
    
    Labelling happens when a pull request first asks for review, then once a day
    over the whole open set. A push can change the tier, but subscribing to every
    push would run this hundreds of times a day to correct a label nobody is
    misled by in the meantime; the daily sweep absorbs that drift and doubles as
    the recovery path for any missed event.
    
    The checkout ref is pinned to the trusted default-branch commit rather than
    left to the event default, since this job imports the checked-out script and
    runs with a writable token.
    
    `scripts/pr-effort-dryrun.mjs` reports the same tiers over live pull requests
    without writing anything, so a boundary or exclusion change is reviewed
    against real data first.
    
    Generated-by: Claude Code
    
    * fix(ci): register the effort label workflow in the trigger allowlist
    
    The CI planner test requires every workflow carrying a pull request trigger
    to be listed explicitly, so adding pr-effort-label.yml without the entry
    failed the `test` job rather than the workflow itself.
    
    Generated-by: Claude Code
    
    * refactor(ci): fold the effort dry run into the workflow
    
    A second script existed only to print what the labelling sweep would do,
    duplicating the sweep to do it. The workflow now takes a dry_run dispatch
    input and logs the plan instead of writing it, so the rehearsal exercises the
    same code path that runs in production rather than a copy that can drift.
    
    planLabels becomes the module's only export for the same reason: the tests
    now bind to the contract the workflow calls, not to helpers that would exist
    only to be tested.
    
    Generated-by: Claude Code
    
    * test(ci): lock the pull_request_target checkout to the base commit
    
    pull_request_target hands the job a writable token while the pull request is
    fork controlled, so the checked-out ref is what decides whether that token can
    reach author-supplied code. copilot-auto-review.yml already stated the rule in
    a comment and pr-effort-label.yml pins the ref, but nothing failed when either
    changed.
    
    The contract covers both files and fails on the two edits that would widen the
    token's reach: pointing ref at head.sha, and dropping ref so a merge-ref event
    supplies the pull request's own tree.
    
    Generated-by: Claude Code
---
 .github/workflows/ci.yml              |   6 ++
 .github/workflows/pr-effort-label.yml | 134 ++++++++++++++++++++++++++++++++++
 scripts/ci-test-plan.test.mjs         |  17 +++++
 scripts/pr-effort.mjs                 |  90 +++++++++++++++++++++++
 scripts/pr-effort.test.mjs            | 132 +++++++++++++++++++++++++++++++++
 5 files changed, 379 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a8e9cc17f..fef8c5a78 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -61,6 +61,12 @@ jobs:
       - name: Test CI planner
         run: node --test --test-concurrency=1 scripts/ci-test-plan.test.mjs scripts/verify-windows-harness.test.mjs
 
+      # Pure Node like the planner test, and the labelling workflow imports this
+      # module directly, so a tier or exclusion change is caught here rather
+      # than by mislabelling live pull requests.
+      - name: Test PR effort classification
+        run: node --test --test-concurrency=1 scripts/pr-effort.test.mjs
+
       # Same shape and the same needs: a regenerate-and-diff contract that runs
       # on Node alone, so it belongs beside the planner test rather than behind
       # an install.
diff --git a/.github/workflows/pr-effort-label.yml b/.github/workflows/pr-effort-label.yml
new file mode 100644
index 000000000..8cd3d71b8
--- /dev/null
+++ b/.github/workflows/pr-effort-label.yml
@@ -0,0 +1,134 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+name: PR effort label
+
+# Labels a pull request when it first asks for review, then re-checks the whole
+# open set once a day. A push changes the diff and so can change the tier, but
+# subscribing to every push would run this hundreds of times a day to correct a
+# label nobody is misled by in the meantime; the daily sweep absorbs that drift
+# and doubles as the recovery path for any event this misses.
+on:
+  pull_request_target:
+    types: [opened, reopened, ready_for_review]
+    branches: [main]
+  schedule:
+    - cron: "23 4 * * *"
+  workflow_dispatch:
+    inputs:
+      dry_run:
+        description: Log the tier each open pull request would get without writing labels.
+        required: false
+        default: false
+        type: boolean
+
+permissions:
+  contents: read
+  pull-requests: write
+
+concurrency:
+  group: pr-effort-label-${{ github.event.pull_request.number || github.workflow }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request_target' }}
+
+jobs:
+  label:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+      # pull_request_target runs with a writable token, so the ref is pinned to
+      # the trusted default-branch commit rather than left to the event's
+      # default: pull_request_review and friends resolve to refs/pull/N/merge,
+      # and this job imports the checked-out script.
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          ref: ${{ github.sha }}
+          persist-credentials: false
+
+      - uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
+        env:
+          DRY_RUN: ${{ inputs.dry_run || false }}
+        with:
+          script: |
+            const path = require("node:path")
+            const { pathToFileURL } = require("node:url")
+
+            const { planLabels } = await import(
+              pathToFileURL(path.join(process.env.GITHUB_WORKSPACE, "scripts/pr-effort.mjs")).href
+            )
+
+            const { owner, repo } = context.repo
+
+            // Reviewing a tier boundary or an exclusion means seeing it against
+            // real pull requests, which is this same sweep minus the writes.
+            const dryRun = process.env.DRY_RUN === "true"
+
+            const targets = context.payload.pull_request
+              ? [context.payload.pull_request.number]
+              : (
+                  await github.paginate(github.rest.pulls.list, {
+                    owner,
+                    repo,
+                    state: "open",
+                    per_page: 100,
+                  })
+                ).map((pull) => pull.number)
+
+            for (const pull_number of targets) {
+              const [files, current] = await Promise.all([
+                github.paginate(github.rest.pulls.listFiles, { owner, repo, pull_number, per_page: 100 }),
+                github.paginate(github.rest.issues.listLabelsOnIssue, {
+                  owner,
+                  repo,
+                  issue_number: pull_number,
+                  per_page: 100,
+                }),
+              ])
+
+              const plan = planLabels(
+                files,
+                current.map((label) => label.name),
+              )
+
+              if (dryRun) {
+                core.info(
+                  `#${pull_number}: ${plan.label} (${plan.lines} readable lines)` +
+                    ` +[${plan.addLabels.join(", ")}] -[${plan.removeLabels.join(", ")}]`,
+                )
+                continue
+              }
+
+              if (plan.addLabels.length > 0) {
+                await github.rest.issues.addLabels({
+                  owner,
+                  repo,
+                  issue_number: pull_number,
+                  labels: plan.addLabels,
+                })
+              }
+
+              for (const name of plan.removeLabels) {
+                try {
+                  await github.rest.issues.removeLabel({ owner, repo, issue_number: pull_number, name })
+                } catch (error) {
+                  // Another run may have removed it first; anything else is real.
+                  if (error.status !== 404) throw error
+                }
+              }
+
+              core.info(`#${pull_number}: ${plan.label} (${plan.lines} readable lines)`)
+            }
diff --git a/scripts/ci-test-plan.test.mjs b/scripts/ci-test-plan.test.mjs
index 5bc8dd511..6eb1886cb 100644
--- a/scripts/ci-test-plan.test.mjs
+++ b/scripts/ci-test-plan.test.mjs
@@ -345,6 +345,7 @@ test('pull request triggers stay on an explicit allowlist', () => {
     'copilot-auto-review.yml',
     'dependency-audit.yml',
     'gitoxide-helper-admission.yml',
+    'pr-effort-label.yml',
     'release-windows-check.yml',
     'runtime-host-owner-platform.yml',
     'runtime-host-peer-admission.yml',
@@ -586,6 +587,22 @@ test('workflows never persist the job credential into the checkout', () => {
   }
 });
 
+test('a pull_request_target checkout is pinned to the trusted base commit', () => {
+  // This event hands the job a writable token while the pull request is fork
+  // controlled, so what gets checked out is what decides whether that token can
+  // reach author-supplied code. `github.sha` is the base branch commit here;
+  // `head.sha` and a bare checkout under a merge-ref event are both the pull
+  // request's own tree. Nothing else in CI would notice that edit, which is why
+  // the rule lives here rather than in a comment.
+  for (const name of readdirSync(WORKFLOW_DIR)) {
+    if (!/\bpull_request_target\b/u.test(triggerBlock(name))) continue;
+
+    for (const step of checkoutSteps(name)) {
+      assert.match(step, /\n\s+ref: \$\{\{ github\.sha \}\}\n/u, `${name}: ${step.trim()}`);
+    }
+  }
+});
+
 test('core CI runs the live Eval proxy lifecycle when Eval is selected', () => {
   const workflow = readWorkflow('ci.yml');
   const evalPackage = JSON.parse(
diff --git a/scripts/pr-effort.mjs b/scripts/pr-effort.mjs
new file mode 100644
index 000000000..ea155f204
--- /dev/null
+++ b/scripts/pr-effort.mjs
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// How much reading a pull request asks for. Review state is deliberately not
+// labelled here: GitHub already indexes review, check and draft state, so a
+// search qualifier answers "whose move is it" without a workflow keeping a
+// copy fresh. Pull request search has no size qualifier, which leaves reading
+// effort as the one axis a query cannot express.
+
+const EFFORT_LABELS = ['effort/XS', 'effort/S', 'effort/M', 'effort/L', 'effort/XL'];
+
+// Counted changes should track what a human actually reads. Lockfiles,
+// regenerated artifacts and binaries are verified by their own contracts, so
+// letting their line counts reach the tiers would inflate every pull request
+// that happens to touch one. A Rust dependency bump alone rewrites thousands
+// of Cargo.lock lines that nobody reviews line by line.
+const UNREAD_PATTERNS = [
+  /(^|\/)(package-lock\.json|pnpm-lock\.yaml|yarn\.lock|Cargo\.lock|uv\.lock)$/,
+  /(^|\/)THIRD_PARTY_(NOTICES|LICENSES)[^/]*$/,
+  /\.generated\.[cm]?[jt]sx?$/,
+  /\.snapshot\.json$/,
+  /\.min\.(js|css)$/,
+  /\.(png|jpe?g|gif|ico|webp|woff2?|ttf|sqlite|zip|gz|pdf)$/,
+];
+
+// Tier boundaries are inclusive upper bounds on readable lines. Test code is
+// not discounted anywhere here; it is reviewed too.
+const EFFORT_TIERS = [
+  { label: 'effort/XS', maxLines: 10 },
+  { label: 'effort/S', maxLines: 100 },
+  { label: 'effort/M', maxLines: 500 },
+  { label: 'effort/L', maxLines: 1000 },
+  { label: 'effort/XL', maxLines: Number.POSITIVE_INFINITY },
+];
+
+function isUnreadPath(path) {
+  const normalized = String(path).replace(/\\/g, '/');
+  return UNREAD_PATTERNS.some((pattern) => pattern.test(normalized));
+}
+
+/**
+ * @param {Array<{filename: string, additions?: number, deletions?: number}>} files
+ */
+function countReadableLines(files = []) {
+  return files.reduce((total, file) => {
+    if (isUnreadPath(file.filename)) return total;
+    return total + (file.additions ?? 0) + (file.deletions ?? 0);
+  }, 0);
+}
+
+/**
+ * @param {Array<object>} files
+ */
+function classifyEffort(files = []) {
+  const lines = countReadableLines(files);
+  const tier = EFFORT_TIERS.find((candidate) => lines <= candidate.maxLines);
+  return { label: tier.label, lines };
+}
+
+/**
+ * @param {Array<object>} files
+ * @param {string[]} currentLabels
+ */
+export function planLabels(files = [], currentLabels = []) {
+  const { label, lines } = classifyEffort(files);
+  const current = new Set(currentLabels);
+
+  return {
+    label,
+    lines,
+    addLabels: current.has(label) ? [] : [label],
+    removeLabels: EFFORT_LABELS.filter((name) => name !== label && current.has(name)),
+  };
+}
diff --git a/scripts/pr-effort.test.mjs b/scripts/pr-effort.test.mjs
new file mode 100644
index 000000000..d7913f5a9
--- /dev/null
+++ b/scripts/pr-effort.test.mjs
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+// Exercised through planLabels, the module's only export, so the tests bind to
+// the contract the workflow calls rather than to helpers that would then exist
+// only to be tested.
+
+import assert from 'node:assert/strict';
+import { describe, it } from 'node:test';
+
+import { planLabels } from './pr-effort.mjs';
+
+function file(filename, additions, deletions = 0) {
+  return { filename, additions, deletions };
+}
+
+function tier(files) {
+  return planLabels(files).label;
+}
+
+function readable(files) {
+  return planLabels(files).lines;
+}
+
+describe('tier boundaries', () => {
+  it('places each tier on its inclusive upper bound', () => {
+    assert.equal(tier([file('a.ts', 10)]), 'effort/XS');
+    assert.equal(tier([file('a.ts', 11)]), 'effort/S');
+    assert.equal(tier([file('a.ts', 100)]), 'effort/S');
+    assert.equal(tier([file('a.ts', 101)]), 'effort/M');
+    assert.equal(tier([file('a.ts', 500)]), 'effort/M');
+    assert.equal(tier([file('a.ts', 501)]), 'effort/L');
+    assert.equal(tier([file('a.ts', 1000)]), 'effort/L');
+    assert.equal(tier([file('a.ts', 1001)]), 'effort/XL');
+  });
+
+  it('counts additions and deletions together', () => {
+    assert.equal(readable([file('a.ts', 40, 2)]), 42);
+  });
+
+  it('treats an empty pull request as the smallest tier', () => {
+    assert.equal(tier([]), 'effort/XS');
+  });
+});
+
+describe('unread paths', () => {
+  it('excludes every lockfile format in the tree', () => {
+    assert.equal(readable([file('pnpm-lock.yaml', 9000, 8000)]), 0);
+    assert.equal(readable([file('package-lock.json', 9000)]), 0);
+    assert.equal(readable([file('native/runtime-host-peer/Cargo.lock', 3524, 100)]), 0);
+    assert.equal(readable([file('uv.lock', 500)]), 0);
+  });
+
+  it('excludes both spellings the notice generator emits', () => {
+    assert.equal(
+      readable([file('apps/desktop/resources/licenses/npm/THIRD_PARTY_NOTICES.txt', 900)]),
+      0,
+    );
+    assert.equal(
+      readable([file('apps/desktop/src/renderer/public/THIRD_PARTY_LICENSES.txt', 900)]),
+      0,
+    );
+  });
+
+  it('excludes generated sources, snapshots and binaries', () => {
+    assert.equal(readable([file('packages/core/src/model-metadata.generated.ts', 5000)]), 0);
+    assert.equal(readable([file('scripts/model-metadata/models-dev-api.snapshot.json', 4000)]), 0);
+    assert.equal(readable([file('packages/storage/test-fixtures/v0.1.6/runtime.sqlite', 1)]), 0);
+    assert.equal(readable([file('apps/desktop/build/background@2x.png', 1)]), 0);
+  });
+
+  it('normalizes Windows separators', () => {
+    assert.equal(readable([file('native\\runtime-host-peer\\Cargo.lock', 3524)]), 0);
+  });
+
+  it('keeps hand-authored sources, including tests and locale copy', () => {
+    assert.equal(readable([file('packages/cli/src/main.ts', 20, 4)]), 24);
+    assert.equal(readable([file('packages/core/test/session.test.ts', 300)]), 300);
+    assert.equal(readable([file('apps/desktop/src/renderer/locales/mcp-copy.ts', 80)]), 80);
+    // Named for the notices but hand-written policy prose.
+    assert.equal(readable([file('docs/third-party-notices-policy.md', 40)]), 40);
+  });
+
+  it('keeps a dependency bump at the tier its readable diff earns', () => {
+    assert.equal(
+      tier([file('pnpm-lock.yaml', 5000, 4000), file('package.json', 2, 2)]),
+      'effort/XS',
+    );
+  });
+});
+
+describe('label planning', () => {
+  it('adds the tier and leaves unrelated labels alone', () => {
+    const plan = planLabels([file('a.ts', 5)], ['bug', 'area/cli']);
+    assert.deepEqual(plan.addLabels, ['effort/XS']);
+    assert.deepEqual(plan.removeLabels, []);
+  });
+
+  it('replaces a tier the pull request has outgrown', () => {
+    const plan = planLabels([file('a.ts', 2000)], ['effort/S', 'bug']);
+    assert.deepEqual(plan.addLabels, ['effort/XL']);
+    assert.deepEqual(plan.removeLabels, ['effort/S']);
+  });
+
+  it('collapses duplicate tiers left by an earlier run', () => {
+    const plan = planLabels([file('a.ts', 5)], ['effort/XS', 'effort/M', 'effort/XL']);
+    assert.deepEqual(plan.addLabels, []);
+    assert.deepEqual(plan.removeLabels, ['effort/M', 'effort/XL']);
+  });
+
+  it('is idempotent once the tier already matches', () => {
+    const plan = planLabels([file('a.ts', 5)], ['effort/XS']);
+    assert.deepEqual(plan.addLabels, []);
+    assert.deepEqual(plan.removeLabels, []);
+  });
+});


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@maka.apache.org
For additional commands, e-mail: commits-help@maka.apache.org