You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/07/10 09:57:36 UTC
(magpie) branch main updated: Align pr-management-stats specs with the fold-channel triage fix (#799)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new 4e217be0 Align pr-management-stats specs with the fold-channel triage fix (#799)
4e217be0 is described below
commit 4e217be05cad0cf887b72523a63e6d3cc43f4222
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Fri Jul 10 11:57:30 2026 +0200
Align pr-management-stats specs with the fold-channel triage fix (#799)
PR #684 fixed the stats reference implementation to count pr-body folded
triage notes across both feedback channels, but the prose specs that
adopter agents actually follow (the tools/ reference is not shipped in the
snapshot) were left describing the old comment-only behaviour. An agent
following aggregate.md/classify.md therefore reproduces the very bug #684
fixed: recent-week Triage velocity reads ~0 because almost all triage is
folded into the PR body, and is_ai_triaged over-counts because it was
documented as an independent any-AI-footer-comment predicate.
Bring the docs in line with triage_marker_events:
- Triage velocity now takes the earliest triage marker across BOTH channels
(QC comment or pr-triage-fold triaged= timestamp), with the comment-cap
caveat scoped to the comment channel only.
- is_ai_triaged routes through the triage markers: a fold block is always
AI-drafted (detected structurally, not by a footer substring, since the
fold footer text differs), making is_ai_triaged a subset of is_triaged.
- Triager-activity AI/manual split credits the fold channel to the operator
named in the fold header.
Co-authored-by: Tester <t...@example.com>
---
skills/pr-management-stats/aggregate.md | 52 +++++++++++++++--------
skills/pr-management-stats/classify.md | 73 ++++++++++++++++++++-------------
2 files changed, 80 insertions(+), 45 deletions(-)
diff --git a/skills/pr-management-stats/aggregate.md b/skills/pr-management-stats/aggregate.md
index 25764aab..f16dffa7 100644
--- a/skills/pr-management-stats/aggregate.md
+++ b/skills/pr-management-stats/aggregate.md
@@ -297,6 +297,13 @@ for each comment c by login L where authorAssociation IN
if bucket is one of the 6 windows:
kind = "ai" if "AI-assisted triage tool" in c.body else "manual"
triager_weekly[L][bucket][kind] += (first-engagement-in-bucket counter)
+
+# ALSO credit the body-fold channel (the default) — a comment-only walk misses it:
+for each PR with a `pr-triage-fold` block:
+ operator = login parsed from the fold header (``by `@<login>` ``)
+ bucket = which_week(fold `triaged=<ISO>` timestamp)
+ if operator is a non-bot maintainer AND bucket is one of the 6 windows:
+ triager_weekly[operator][bucket]["ai"] += 1 # a fold is always AI-drafted
```
Per-week counting rule: count **at most one PR per (login, week, kind)** even
@@ -311,9 +318,11 @@ PR of the same kind collapse to a single tally.
Every triager's per-week count is further split into:
- **AI-assisted**: comments whose body contains the AI-attribution footer
- substring (`AI-assisted triage tool` — same detector as
- [`is_ai_triaged`](classify.md#is_ai_triaged--ai-assisted-triage)).
-- **Manual**: comments without the footer.
+ substring (`AI-assisted triage tool` — same comment-channel detector as
+ [`is_ai_triaged`](classify.md#is_ai_triaged--ai-assisted-triage)), **plus every
+ `pr-triage-fold` note**, which is always AI-drafted and credited to the
+ maintainer named in the fold header.
+- **Manual**: comment-channel engagement without the AI footer.
Same PR can contribute to *both* sub-counts for the same maintainer-week if
they posted both an AI-drafted comment and a manually-typed comment in the
@@ -419,25 +428,34 @@ drift.
## Triage velocity (per-week)
-The "Triage velocity" panel counts PRs whose **first** Quality-Criteria marker
-comment fell in each week. Distinct from the closure-by-reason chart (which
-counts when PRs *closed*) — this counts when triage *happened*.
+The "Triage velocity" panel counts PRs whose **first** triage marker fell in each
+week. Distinct from the closure-by-reason chart (which counts when PRs *closed*) —
+this counts when triage *happened*.
-For each PR (open OR closed-since-cutoff), `triage_comment_ts` is the timestamp
-of its *earliest* maintainer comment containing the `Pull Request quality
-criteria` marker. Bucket by week of `triage_comment_ts` (skip PRs that were
-never triaged).
+For each PR (open OR closed-since-cutoff), `triage_ts` is the timestamp of its
+*earliest* triage marker **across both feedback channels** — take `min` over
+[`triage_marker_events(pr)`](classify.md#is_ai_triaged--ai-assisted-triage), which
+yields one event per QC-marker comment (comment channel) **and** one for the
+`pr-triage-fold` block's `triaged=<ISO>` timestamp (body-fold channel, the
+default). Bucket by week of `triage_ts` (skip PRs never triaged in either
+channel). **Counting only the comment channel here is the classic bug**: under the
+default body-fold channel almost every triage is folded into the PR body, so a
+comment-only scan reports recent-week velocity as ≈0 even while triage is running
+at full pace.
-Split each week's count by AI vs manual:
+Split each week's count by AI vs manual — using the `is_ai` flag on the earliest
+triage event (see [`is_ai_triaged`](classify.md#is_ai_triaged--ai-assisted-triage)):
| Series | Definition |
|---|---|
-| `ai` | The triage-comment also contains the [`is_ai_triaged`](classify.md#is_ai_triaged--ai-assisted-triage) footer substring |
-| `non_ai_qc` | Triage comment present but no AI footer (a maintainer pasted the QC link manually) |
-
-⚠ **Data caveat**: the closed-PR fetch caps comments at `last:25` per PR. PRs
-whose first triage marker is older than the 25 most recent comments are
-missed — so older weeks systematically under-count. Note this in the dashboard.
+| `ai` | The earliest triage event is AI-drafted — a comment carrying the AI footer, **or** any `pr-triage-fold` block (a fold is always AI-drafted) |
+| `non_ai_qc` | The earliest triage event is a manual QC-marker comment (no AI footer, no fold) — a maintainer pasted the QC link by hand |
+
+⚠ **Data caveat**: the closed-PR fetch caps comments at `last:25` per PR, so a
+first triage marker delivered as a **comment** older than the 25 most recent
+comments is missed and older weeks under-count on the comment channel. The
+body-fold `triaged=` timestamp lives in the PR body (always fetched), so
+fold-channel velocity is **not** subject to the comment cap.
---
diff --git a/skills/pr-management-stats/classify.md b/skills/pr-management-stats/classify.md
index 2c4fc217..adcfbd91 100644
--- a/skills/pr-management-stats/classify.md
+++ b/skills/pr-management-stats/classify.md
@@ -79,36 +79,49 @@ The terminology in the dashboard:
```text
is_ai_triaged(pr) :=
- EXISTS comment c IN pr.comments
- WHERE c.authorAssociation IN (OWNER, MEMBER, COLLABORATOR)
- AND c.body CONTAINS "AI-assisted triage tool"
- OR pr.body CONTAINS "AI-assisted triage tool" # body-fold footer (default channel)
+ EXISTS triage_event e IN triage_marker_events(pr) WHERE e.is_ai
+ # triage_marker_events(pr) yields one (timestamp, is_ai) per triage marker across BOTH channels:
+ # - comment channel: each OWNER/MEMBER/COLLABORATOR comment whose body contains the
+ # "Pull Request quality criteria" marker; is_ai = comment body CONTAINS "AI-assisted triage tool"
+ # - body-fold channel: the PR body's `pr-triage-fold` block; is_ai = True ALWAYS (a fold is
+ # written by the automated triage tool, so its presence — not a footer substring — is the signal)
```
-A PR received at least one maintainer comment whose body contains the
-**AI-attribution footer substring** (`AI-assisted triage tool`). Every
-`pr-management-triage` comment template (draft / comment / ping /
-request-author-confirmation / close-comment etc.) ends with this footer
-verbatim — so the detector counts any PR whose triage included a skill-drafted
-comment.
-
-This predicate is **independent of** `is_triaged` and `is_engaged`:
-
-- An AI-drafted draft+comment includes the `Pull Request quality criteria`
- link → both `is_triaged` and `is_ai_triaged` fire.
-- An AI-drafted **ping** or **request-author-confirmation** uses a different
- template that does NOT include the criteria link → `is_engaged` and
- `is_ai_triaged` fire but `is_triaged` does NOT.
-- A maintainer's hand-typed comment in the criteria-template form (rare —
- this happens when a maintainer manually pastes the link text without the
- skill) → `is_triaged` and `is_engaged` fire but `is_ai_triaged` does NOT.
+A PR is AI-triaged when a skill-drafted triage marker reached it through **either
+feedback channel**, routed through the single `triage_marker_events` helper so the
+default body-fold channel is never invisible:
+
+- **Comment channel** — an `OWNER`/`MEMBER`/`COLLABORATOR` comment that carries the
+ `Pull Request quality criteria` marker *and* whose body contains the
+ **AI-attribution footer substring** (`AI-assisted triage tool`). The
+ violations-style comment templates end with this footer verbatim.
+- **Body-fold channel** (the default `triage_feedback_channel: pr-body`) — the PR
+ description carries a `pr-triage-fold` block. **A fold block is always
+ AI-drafted** (it is written by the automated triage tool), so its *presence* is
+ the signal. Do **not** match a footer substring against the fold body: the
+ fold's visible footer varies by template (e.g. the ping /
+ request-author-confirmation folds read `Automated triage — may be imperfect`,
+ not `AI-assisted triage tool`), so a substring scan would silently miss them.
+
+Because both cases are triage markers, **`is_ai_triaged` is a subset of
+`is_triaged`** — every AI-drafted triage note is, by construction, a triage note:
+
+- An AI-drafted violations draft/comment (comment channel) carries the
+ `Pull Request quality criteria` link **and** the AI footer → both `is_triaged`
+ and `is_ai_triaged` fire.
+- Any `pr-triage-fold` block (body-fold channel — draft / comment / ping /
+ request-author-confirmation alike) → both `is_triaged` and `is_ai_triaged` fire
+ (a fold is always AI-drafted).
+- A maintainer's hand-typed QC-marker comment with no AI footer → `is_triaged`
+ and `is_engaged` fire but `is_ai_triaged` does NOT (that one triage event is
+ manual).
The implication chains:
```text
-is_triaged(pr) ⇒ is_engaged(pr) # marker requires maintainer comment, which requires engagement
-is_ai_triaged(pr) ⇒ is_engaged(pr) # AI footer requires maintainer comment, which requires engagement
-is_triaged(pr) ⇎ is_ai_triaged(pr) # independent — see cases above
+is_triaged(pr) ⇒ is_engaged(pr) # marker requires maintainer comment/fold, which requires engagement
+is_ai_triaged(pr) ⇒ is_triaged(pr) # every AI-drafted marker is a marker (subset)
+is_ai_triaged(pr) ⇒ is_engaged(pr) # transitively
```
### `is_untriaged` — *broad untriaged*
@@ -149,7 +162,7 @@ The age uses the `last_author_interaction` defined in the
|---|---|---|---|
| Quality-Criteria-triaged | `is_triaged` | maintainer posted the literal `Pull Request quality criteria` link | **Quality Criteria triaged** (hero row 2, blue) |
| De-facto triaged | `is_engaged AND NOT is_triaged` | maintainer engaged but no marker | **De-facto triaged** (hero row 2, amber — the gap signal) |
-| AI-triaged | `is_ai_triaged` | comment with the AI-attribution footer | **AI-triaged** (hero row 2, purple — accounting) |
+| AI-triaged | `is_ai_triaged` | an AI-drafted triage marker — a comment with the AI footer, or any `pr-triage-fold` block | **AI-triaged** (hero row 2, purple — accounting) |
| Engaged (overall) | `is_engaged` | union of the above two | not a card on its own; equals `triaged + defacto_triaged` |
| Untriaged | `is_untriaged` | NOT engaged + contributor + non-bot + not ready-labelled | **Untriaged non-drafts** (hero row 1) |
@@ -268,7 +281,8 @@ Splitting them lets the maintainer focus stale-sweep / ping efforts on the high-
### Caveats
- Both predicates rely on `pr.comments(last:10)` — older outstanding comments on chatty PRs may be missed. Lower-bound numbers.
-- The footer-substring match (`AI-assisted triage tool`) is configurable per [`<project-config>/pr-management-config.md`](../../projects/_template/pr-management-config.md)'s `ai_attribution_substring`. The default works as long as the project doesn't customise the footer text.
+- The footer-substring match (`AI-assisted triage tool`) applies to the **comment
+ channel** only and is configurable per [`<project-config>/pr-management-config.md`](../../projects/_template/pr-management-config.md)'s `ai_attribution_substring`. The **body-fold channel** is detected structurally by the `pr-triage-fold` block (always AI-drafted) and does not depend on this substring.
---
@@ -364,8 +378,11 @@ the dashboard surface the fuller picture.
For the configurable AI-attribution detection substring, adopters can override
[`<project-config>/pr-management-config.md`](../../projects/_template/pr-management-config.md)'s
`ai_attribution_substring` field; the framework defaults to
-`AI-assisted triage tool`. The literal substring is a single point of failure
-— keep it identical between the comment templates and this detector.
+`AI-assisted triage tool`. That substring is the **comment-channel** signal and
+is a single point of failure — keep it identical between the comment templates and
+this detector. The **body-fold channel** does not use it: a `pr-triage-fold` block
+is always AI-drafted and is detected by the marker itself, so no footer text needs
+to stay in sync there.
---