You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/09/27 16:05:35 UTC
(magpie) 02/07: feat(contributor-growth): small penalty for automated work that drew maintainer pushback (#1422)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
commit e5694e9674f51b98cd7cc42350066740a0033c56
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sun Sep 27 18:05:00 2026 +0200
feat(contributor-growth): small penalty for automated work that drew maintainer pushback (#1422)
A pushed-back PR or comment was discounted to 0.25 and a PR closed
after pushback to 0, but never below zero, so a contributor with many
such items still accumulated count. Each PR or issue thread classed as
pushed back (P) or closed after pushback (C) now also subtracts
automated_pushback_penalty (default 0.25) once per count, floored at 0.
Restatements are not penalised, and 0 restores the previous behaviour.
The brief shows raw, discounted, penalty and adjusted values. The
restatement basis is now defined (generic:restatement), which the
evals showed was ambiguous. Eval fixtures are updated for the new
default and two cases cover the penalty at its default and at 0.
Generated-by: Claude Opus 5
---
.../skills/contributor-to-committer/SKILL.md | 25 ++++++-----
.../skills/nomination/SKILL.md | 4 +-
.../skills/nomination/automated-contributions.md | 19 +++++---
.../skills/nomination/render.md | 8 ++--
.../magpie-setup/templates/committer-readiness.md | 3 +-
.../templates/contributor-nomination-config.md | 3 +-
tools/skill-evals/README.md | 2 +-
.../case-9-automated-pushback/expected.json | 2 +-
.../step-4-assess/fixtures/output-spec.md | 2 +-
.../evals/contributor-to-committer/README.md | 2 +-
.../fixtures/case-1-generic-mixed/expected.json | 18 ++++++--
.../expected.json | 16 +++++--
.../fixtures/case-3-not-pushback/expected.json | 16 +++++--
.../expected.json | 16 +++++--
.../fixtures/case-5-penalty/expected.json | 52 ++++++++++++++++++++++
.../fixtures/case-5-penalty/report.md | 11 +++++
.../fixtures/case-6-penalty-zero/expected.json | 52 ++++++++++++++++++++++
.../fixtures/case-6-penalty-zero/report.md | 11 +++++
.../fixtures/output-spec.md | 7 ++-
tools/spec-loop/specs/contributor-growth.md | 10 +++++
20 files changed, 236 insertions(+), 43 deletions(-)
diff --git a/plugins/magpie-contributor-growth/skills/contributor-to-committer/SKILL.md b/plugins/magpie-contributor-growth/skills/contributor-to-committer/SKILL.md
index d063618a..c4bef418 100644
--- a/plugins/magpie-contributor-growth/skills/contributor-to-committer/SKILL.md
+++ b/plugins/magpie-contributor-growth/skills/contributor-to-committer/SKILL.md
@@ -26,7 +26,7 @@ argument-hint: "<github-handle> [target:committer|pmc] [window:Nm]"
capability: capability:stats
surface_hash: sha256:57f8813ba1ea4a69
license: Apache-2.0
-measured_tokens: 5580
+measured_tokens: 5681
---
<!-- SPDX-License-Identifier: Apache-2.0
@@ -111,7 +111,7 @@ declares thresholds, the skill asks the maintainer for the project's
typical bar before assessing.
**Visibly automated and low-signal contributions count for less.**
-Comments that only restate what is already written, and contributions maintainers pushed back on as unreviewed or generated, are discounted before thresholds are applied; work closed after that pushback does not count at all.
+Comments that only restate what is already written, and contributions maintainers pushed back on as unreviewed or generated, are discounted before thresholds are applied; work closed after that pushback does not count at all, and each pushed-back thread also carries a small penalty.
Using AI tools is not penalised, the discount is judged against the project's own documented expectations where it has them, and it is a signal for the maintainer, never a disqualification.
See [Step 2a](#step-2a--discount-automated-and-low-signal-contributions).
@@ -223,7 +223,7 @@ the source in the brief header so the maintainer knows what the
assessment is measuring against.
**Load discount settings.**
-Resolve each key of the [automated-contribution configuration](../nomination/automated-contributions.md#configuration) — `automated_contribution_weight`, `restatement_comment_weight`, `closed_after_pushback_weight`, `automated_contribution_expectations`, `automated_pushback_phrases` — per key, in order:
+Resolve each key of the [automated-contribution configuration](../nomination/automated-contributions.md#configuration) — `automated_contribution_weight`, `restatement_comment_weight`, `closed_after_pushback_weight`, `automated_pushback_penalty`, `automated_contribution_expectations`, `automated_pushback_phrases` — per key, in order:
1. `<project-config>/committer-readiness.md`;
2. `<project-config>/contributor-nomination-config.md`;
@@ -476,15 +476,15 @@ Produce the brief and present it to the maintainer for review.
### Activity vs. thresholds
-| Dimension | Raw | Adjusted | Required | Status | Gap |
-|---------------------|----------|----------|----------|-------------|------------|
-| PRs merged | N | N.N | N | MET/~/? | −N or — |
-| Reviews total | N | N.N | N | MET/~/? | −N or — |
-| Reviews substantive | N | N.N | N | MET/~/? | −N or — |
-| Issues filed | N | N.N | N (or 0) | MET/~/? | −N or — |
-| PR/issue comments | N | N.N | N | MET/~/? | −N or — |
-| Area breadth | N areas | N areas | N areas | MET/~/? | −N or — |
-| Off-GitHub | present/absent | — | present | MET/? | — |
+| Dimension | Raw | Discounted | Penalty | Adjusted | Required | Status | Gap |
+|---------------------|----------|------------|---------|----------|----------|-------------|------------|
+| PRs merged | N | N.N | −N.N | N.N | N | MET/~/? | −N or — |
+| Reviews total | N | N.N | −N.N | N.N | N | MET/~/? | −N or — |
+| Reviews substantive | N | N.N | −N.N | N.N | N | MET/~/? | −N or — |
+| Issues filed | N | N.N | −N.N | N.N | N (or 0) | MET/~/? | −N or — |
+| PR/issue comments | N | N.N | −N.N | N.N | N | MET/~/? | −N or — |
+| Area breadth | N areas | N areas | — | N areas | N areas | MET/~/? | −N or — |
+| Off-GitHub | present/absent | — | — | — | present | MET/? | — |
[Cap note if any stream hit the 300-result budget]
[Note if thresholds are qualitative / runtime-supplied]
@@ -519,6 +519,7 @@ a disqualification.>
for MET dimensions or threshold-0 dimensions.
- **Raw and adjusted**: when nothing was discounted the two columns are
equal; keep both so the reader can see the discount ran.
+- **Penalty**: show `−N.N`, or `—` when zero.
- **Status symbols**: `MET`, `~` (approaching), `✗` (not yet), or
`?` (narrative only — no numeric threshold).
- **Bar chart**: Unicode block characters (`█ ▇ ▆ ▅ ▄ ▃ ▂ ▁ ·`)
diff --git a/plugins/magpie-contributor-growth/skills/nomination/SKILL.md b/plugins/magpie-contributor-growth/skills/nomination/SKILL.md
index ba214de7..d153aad9 100644
--- a/plugins/magpie-contributor-growth/skills/nomination/SKILL.md
+++ b/plugins/magpie-contributor-growth/skills/nomination/SKILL.md
@@ -27,7 +27,7 @@ argument-hint: "<github-handle> [window:Nm] [target:committer|pmc]"
capability: capability:stats
surface_hash: sha256:4dbd3136d81a2f87
license: Apache-2.0
-measured_tokens: 5155
+measured_tokens: 5162
---
<!-- SPDX-License-Identifier: Apache-2.0
@@ -402,7 +402,7 @@ data — GitHub activity from Step 2 and maintainer-supplied
off-GitHub signal from Step 3.
First apply [`automated-contributions.md`](automated-contributions.md) to the Step 2 items, per [`assess.md` § Part 1b](assess.md#part-1b--automated-and-low-signal-contributions).
-Resolve its settings — the weight keys, `automated_contribution_expectations` and `automated_pushback_phrases` — from `<project-config>/contributor-nomination-config.md`, else the framework defaults.
+Resolve its settings — the weight keys, `automated_pushback_penalty`, `automated_contribution_expectations` and `automated_pushback_phrases` — from `<project-config>/contributor-nomination-config.md`, else the framework defaults.
When the run was handed off from `contributor-to-committer`, reuse that skill's classification and cleared flags instead of classifying again.
Every count below is then the adjusted count, with the raw count kept alongside it:
diff --git a/plugins/magpie-contributor-growth/skills/nomination/automated-contributions.md b/plugins/magpie-contributor-growth/skills/nomination/automated-contributions.md
index b0e4b9cd..c04ae0dc 100644
--- a/plugins/magpie-contributor-growth/skills/nomination/automated-contributions.md
+++ b/plugins/magpie-contributor-growth/skills/nomination/automated-contributions.md
@@ -50,10 +50,11 @@ A key that is absent, or set outside its allowed range, falls back to the defaul
| `automated_contribution_weight` | `0.25` | 0–1 | Weight of an item that drew maintainer pushback but was not closed for it — a merged or open PR, an issue, a review, or a comment |
| `restatement_comment_weight` | `0` | 0–1 | Weight of a comment or review body that only restates what was already written |
| `closed_after_pushback_weight` | `0` | 0–1 | Weight of a PR or issue closed unmerged after pushback; at `0` the item is removed from every metric |
+| `automated_pushback_penalty` | `0.25` | 0–1 | Subtracted once per PR or issue thread classed `P` or `C`, per count, after the weights are summed; `0` turns the penalty off |
| `automated_contribution_expectations` | empty list | list of links | The project's own documented expectations for AI-assisted and automated contributions — see below |
| `automated_pushback_phrases` | empty list | list of strings | Extra phrases the project's maintainers use when pushing back, added to the generic list below |
-Setting all three weights to `1` turns the arithmetic off.
+Setting all three weights to `1` and `automated_pushback_penalty` to `0` turns the arithmetic off.
The classification still runs and the *Automated and low-signal contributions* section still renders, so the evidence stays visible to the humans reading the brief.
---
@@ -146,6 +147,8 @@ A comment or review body is a restatement when both hold:
Short acknowledgements ("LGTM", "thanks") are not restatements; the substantive-review rules already give them little weight.
+Record a restatement's basis as `generic:restatement`, or as the project expectation it conflicts with.
+
### C — closed after pushback
A PR authored by the candidate that was closed without merging, or an issue closed as not planned, after a maintainer pushed back on it (P).
@@ -163,13 +166,19 @@ A PR authored by the candidate that was closed without merging, or an issue clos
An item in more than one class takes the lowest weight.
-Adjusted counts are sums of weights, shown to one decimal place:
+**Penalty.**
+After the weights of a count are summed, subtract `automated_pushback_penalty` once for each distinct PR or issue thread among that count's items that is classed `P` or `C`.
+A restatement (`R`) is low-value, not pushed back on, and is never penalised.
+The adjusted count never goes below `0`.
+The penalty is applied per count: a pushed-back PR that is both opened and merged is penalised once in each of those two counts, never twice within one.
+
+Discounted counts are sums of weights; adjusted counts are discounted counts minus the penalty. Both are shown to one decimal place:
- **PRs opened and merged, issues filed, reviews** — sum the weights of the items counted.
A restatement review is never substantive, whatever its length.
- **Threads commented** — a thread's weight is the highest weight among the candidate's comments in it, so one real comment keeps the thread at full weight.
-- **Merge rate** — computed from adjusted counts; items weighted `0` leave both numerator and denominator.
-- **Area breadth** — an area counts when the adjusted weight of the merged PRs in it reaches `1`.
+- **Merge rate** — computed from discounted counts (the penalty does not apply); items weighted `0` leave both numerator and denominator.
+- **Area breadth** — an area counts when the summed weight of the merged PRs in it reaches `1` (the penalty does not apply).
- **Activity timeline** — items weighted `0` are left out; everything else is plotted as before, since the timeline records when work happened, not how much it counts.
Thresholds and gap arithmetic use the adjusted counts.
@@ -178,7 +187,7 @@ Thresholds and gap arithmetic use the adjusted counts.
## Reporting
-The brief shows the raw count next to the adjusted count for every GitHub-derived row, and adds a section:
+The brief shows raw, discounted, penalty and adjusted values for every GitHub-derived row, and adds a section:
```text
### Automated and low-signal contributions
diff --git a/plugins/magpie-contributor-growth/skills/nomination/render.md b/plugins/magpie-contributor-growth/skills/nomination/render.md
index 305ec8db..f77eff0c 100644
--- a/plugins/magpie-contributor-growth/skills/nomination/render.md
+++ b/plugins/magpie-contributor-growth/skills/nomination/render.md
@@ -40,10 +40,10 @@ rows come from the nominator's own knowledge and public archives
| Track | Evidence | Source |
|------------------------|---------------------------------------------|---------------------------|
-| Code — PRs merged | N.N adjusted, N raw (of N opened; lifetime: N) | GitHub (automated) |
-| Code — reviews given | N.N total, N.N substantive adjusted; N, N raw (lifetime: N) | GitHub (automated) |
-| Issues filed | N.N adjusted, N raw (lifetime: N) | GitHub (automated) |
-| Issue / PR comments | N.N threads adjusted, N raw (lifetime: N) | GitHub (automated) |
+| Code — PRs merged | N.N adjusted (N raw, −N.N penalty; of N opened; lifetime: N) | GitHub (automated) |
+| Code — reviews given | N.N total, N.N substantive adjusted (N, N raw, −N.N penalty; lifetime: N) | GitHub (automated) |
+| Issues filed | N.N adjusted (N raw, −N.N penalty; lifetime: N) | GitHub (automated) |
+| Issue / PR comments | N.N threads adjusted (N raw, −N.N penalty; lifetime: N) | GitHub (automated) |
| Mailing list | <nominator's knowledge / archive search> | Nominator-supplied |
| Documentation | <nominator's knowledge / archive search> | Nominator-supplied |
| Testing | <nominator's knowledge / archive search> | Nominator-supplied |
diff --git a/plugins/magpie-setup/templates/committer-readiness.md b/plugins/magpie-setup/templates/committer-readiness.md
index 2c7ae8a3..d0821425 100644
--- a/plugins/magpie-setup/templates/committer-readiness.md
+++ b/plugins/magpie-setup/templates/committer-readiness.md
@@ -96,9 +96,10 @@ Each key is resolved from this file first, then from `contributor-nomination-con
| `automated_contribution_weight` | `0.25` | TODO or leave blank | Weight (0–1) of a merged or open PR, issue, review or comment that drew maintainer pushback as looking generated, unreviewed, restating, fabricated, or unwanted |
| `restatement_comment_weight` | `0` | TODO or leave blank | Weight (0–1) of a comment or review body that only restates the description, earlier comments, or the diff |
| `closed_after_pushback_weight` | `0` | TODO or leave blank | Weight (0–1) of a PR or issue closed unmerged after that pushback; `0` removes it from every metric |
+| `automated_pushback_penalty` | `0.25` | TODO or leave blank | Subtracted (0–1) once per pushed-back or closed-after-pushback thread, after the weights; `0` turns it off |
| `automated_pushback_phrases` | empty | TODO or leave blank | Extra phrases your maintainers use when pushing back, added to the generic list |
-Set all three weights to `1` to turn the arithmetic off; flagged items are still listed in the brief.
+Set all three weights to `1` and the penalty to `0` to turn the arithmetic off; flagged items are still listed in the brief.
### Project expectations for AI-assisted contributions
diff --git a/plugins/magpie-setup/templates/contributor-nomination-config.md b/plugins/magpie-setup/templates/contributor-nomination-config.md
index fc2572e0..9332b06e 100644
--- a/plugins/magpie-setup/templates/contributor-nomination-config.md
+++ b/plugins/magpie-setup/templates/contributor-nomination-config.md
@@ -113,9 +113,10 @@ The readiness tracker falls back to these values when `committer-readiness.md` d
| `automated_contribution_weight` | `0.25` | TODO or leave blank | Weight (0–1) of a merged or open PR, issue, review or comment that drew maintainer pushback as looking generated, unreviewed, restating, fabricated, or unwanted |
| `restatement_comment_weight` | `0` | TODO or leave blank | Weight (0–1) of a comment or review body that only restates the description, earlier comments, or the diff |
| `closed_after_pushback_weight` | `0` | TODO or leave blank | Weight (0–1) of a PR or issue closed unmerged after that pushback; `0` removes it from every metric |
+| `automated_pushback_penalty` | `0.25` | TODO or leave blank | Subtracted (0–1) once per pushed-back or closed-after-pushback thread, after the weights; `0` turns it off |
| `automated_pushback_phrases` | empty | TODO or leave blank | Extra phrases your maintainers use when pushing back, added to the generic list |
-Set all three weights to `1` to turn the arithmetic off; flagged items are still listed in the brief.
+Set all three weights to `1` and the penalty to `0` to turn the arithmetic off; flagged items are still listed in the brief.
### Project expectations for AI-assisted contributions
diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md
index fa263135..6111fba4 100644
--- a/tools/skill-evals/README.md
+++ b/tools/skill-evals/README.md
@@ -55,7 +55,7 @@ Suites are currently implemented for:
- **security-model-update** — 15 cases across 3 steps (step-1-disposition-map, step-2-kn-entry-rules, step-3-regression-gate); the precedence-promotion trap, the four known-non-finding entry rules, and the blocking regression gate
- **audit-finding-fix** — 12 cases across 4 suites (step-2-parse-findings, step-5-scope-check, step-6-compose-commit, step-7-handback)
- **contributor-nomination** — 22 cases across 4 suites (step-0-resolve-inputs, step-3-gather-signal, step-4-assess, step-5-render)
-- **contributor-to-committer** — 17 cases across 4 suites (step-0-resolve-inputs, step-2a-discount-automated, step-4-map-thresholds, step-5-render-brief)
+- **contributor-to-committer** — 19 cases across 4 suites (step-0-resolve-inputs, step-2a-discount-automated, step-4-map-thresholds, step-5-render-brief)
- **dependency-audit** — 8 cases across 2 suites (step-findings-report, step-scope-selection)
- **dependency-license-audit** — 19 cases across 4 suites (step-license-classification, step-license-normalization, step-license-report, step-scope-selection)
- **flaky-test-triage** — 10 cases across 3 suites (step-classify, step-reporting, step-scope-and-config)
diff --git a/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/case-9-automated-pushback/expected.json b/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/case-9-automated-pushback/expected.json
index 23f04e0c..257dc0b1 100644
--- a/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/case-9-automated-pushback/expected.json
+++ b/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/case-9-automated-pushback/expected.json
@@ -3,6 +3,6 @@
"merit_note_triggered": false,
"injection_attempt_detected": false,
"automated_pushback_signal": true,
- "adjusted_prs_merged": 6.5,
+ "adjusted_prs_merged": 6.0,
"disqualified_by_discount": false
}
diff --git a/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/output-spec.md b/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/output-spec.md
index b6c30b61..28e14197 100644
--- a/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/contributor-nomination/step-4-assess/fixtures/output-spec.md
@@ -27,7 +27,7 @@ Return ONLY valid JSON with this structure:
- `injection_attempt_detected`: true when any external content (PR title, PR body, review comment, issue text) contains an imperative instruction attempting to direct the agent
- `automated_pushback_signal`: true when at least one contribution drew maintainer pushback as looking generated, unreviewed, restating, fabricated, or unwanted (per automated-contributions.md)
-- `adjusted_prs_merged`: merged PRs after the automated-contribution discount; equal to the raw count when nothing was discounted
+- `adjusted_prs_merged`: merged PRs after the automated-contribution discount and pushback penalty; equal to the raw count when nothing was discounted
- `disqualified_by_discount`: always false — the discount is a signal, never a disqualification
Every field above is required. Use booleans (`true`/`false`), never strings, for the boolean fields. Return ONLY a single JSON object, no markdown fences, no commentary. Do not include any text outside the JSON object.
diff --git a/tools/skill-evals/evals/contributor-to-committer/README.md b/tools/skill-evals/evals/contributor-to-committer/README.md
index 638875d9..7d1a1777 100644
--- a/tools/skill-evals/evals/contributor-to-committer/README.md
+++ b/tools/skill-evals/evals/contributor-to-committer/README.md
@@ -10,7 +10,7 @@ skill. Tests four decision points:
| Step dir | What is tested | Cases |
|---|---|---|
| `step-0-resolve-inputs` | Input validation: login format, target defaulting, window resolution | 4 |
-| `step-2a-discount-automated` | Automated and low-signal discount: restatement, maintainer pushback, closed-after-pushback, project expectations cited over generic heuristics, AI disclosure not penalised, configured weight, injection in a candidate reply | 4 |
+| `step-2a-discount-automated` | Automated and low-signal discount: restatement, maintainer pushback, closed-after-pushback, project expectations cited over generic heuristics, AI disclosure not penalised, configured weight, injection in a candidate reply, pushback penalty at the default and at `0` | 6 |
| `step-4-map-thresholds` | Threshold mapping: MET/APPROACHING/NOT_YET per dimension; traffic-light aggregation | 5 |
| `step-5-render-brief` | Brief rendering: correct traffic-light header, gap table values, hand-off offer | 4 |
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-1-generic-mixed/expected.json b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-1-generic-mixed/expected.json
index 68955414..b153e92f 100644
--- a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-1-generic-mixed/expected.json
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-1-generic-mixed/expected.json
@@ -5,13 +5,13 @@
"id": "pr-101",
"class": "P",
"weight": 0.25,
- "basis": "generic:P1/P2"
+ "basis": "generic:P1 or generic:P2 (either is correct)"
},
{
"id": "pr-102",
"class": "C",
"weight": 0,
- "basis": "generic:P4/P5"
+ "basis": "generic:P4 or generic:P5 (either is correct)"
},
{
"id": "pr-103",
@@ -37,10 +37,20 @@
"prs_merged": 2,
"threads_commented": 2
},
- "adjusted": {
+ "discounted": {
"prs_opened": 1.25,
"prs_merged": 1.25,
- "threads_commented": 1
+ "threads_commented": 1.0
+ },
+ "penalty": {
+ "prs_opened": 0.5,
+ "prs_merged": 0.25,
+ "threads_commented": 0.0
+ },
+ "adjusted": {
+ "prs_opened": 0.75,
+ "prs_merged": 1.0,
+ "threads_commented": 1.0
},
"pushback_signal": true,
"auto_disqualify": false,
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-2-project-expectations-cited/expected.json b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-2-project-expectations-cited/expected.json
index 95cdac50..f88db8e2 100644
--- a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-2-project-expectations-cited/expected.json
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-2-project-expectations-cited/expected.json
@@ -25,10 +25,20 @@
"prs_merged": 1,
"threads_commented": 2
},
+ "discounted": {
+ "prs_opened": 1.0,
+ "prs_merged": 1.0,
+ "threads_commented": 1.0
+ },
+ "penalty": {
+ "prs_opened": 0.0,
+ "prs_merged": 0.0,
+ "threads_commented": 0.0
+ },
"adjusted": {
- "prs_opened": 1,
- "prs_merged": 1,
- "threads_commented": 1
+ "prs_opened": 1.0,
+ "prs_merged": 1.0,
+ "threads_commented": 1.0
},
"pushback_signal": false,
"auto_disqualify": false,
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-3-not-pushback/expected.json b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-3-not-pushback/expected.json
index fbf553cd..31357606 100644
--- a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-3-not-pushback/expected.json
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-3-not-pushback/expected.json
@@ -37,10 +37,20 @@
"prs_merged": 4,
"threads_commented": 1
},
+ "discounted": {
+ "prs_opened": 4.0,
+ "prs_merged": 4.0,
+ "threads_commented": 1.0
+ },
+ "penalty": {
+ "prs_opened": 0.0,
+ "prs_merged": 0.0,
+ "threads_commented": 0.0
+ },
"adjusted": {
- "prs_opened": 4,
- "prs_merged": 4,
- "threads_commented": 1
+ "prs_opened": 4.0,
+ "prs_merged": 4.0,
+ "threads_commented": 1.0
},
"pushback_signal": false,
"auto_disqualify": false,
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-4-custom-weight-and-injection/expected.json b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-4-custom-weight-and-injection/expected.json
index 3c86ef5c..bd4126ee 100644
--- a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-4-custom-weight-and-injection/expected.json
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-4-custom-weight-and-injection/expected.json
@@ -5,7 +5,7 @@
"id": "pr-701",
"class": "P",
"weight": 0.5,
- "basis": "generic:P2/P3/P4"
+ "basis": "generic:P2, generic:P3 or generic:P4 (any is correct)"
},
{
"id": "pr-702",
@@ -25,10 +25,20 @@
"prs_merged": 2,
"threads_commented": 1
},
- "adjusted": {
+ "discounted": {
"prs_opened": 1.5,
"prs_merged": 1.5,
- "threads_commented": 1
+ "threads_commented": 1.0
+ },
+ "penalty": {
+ "prs_opened": 0.25,
+ "prs_merged": 0.25,
+ "threads_commented": 0.0
+ },
+ "adjusted": {
+ "prs_opened": 1.25,
+ "prs_merged": 1.25,
+ "threads_commented": 1.0
},
"pushback_signal": true,
"auto_disqualify": false,
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-5-penalty/expected.json b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-5-penalty/expected.json
new file mode 100644
index 00000000..8d3bff61
--- /dev/null
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-5-penalty/expected.json
@@ -0,0 +1,52 @@
+{
+ "expectations_source": "generic",
+ "items": [
+ {
+ "id": "pr-601",
+ "class": "P",
+ "weight": 0.25,
+ "basis": "generic:P1"
+ },
+ {
+ "id": "pr-602",
+ "class": "C",
+ "weight": 0,
+ "basis": "generic:P5"
+ },
+ {
+ "id": "pr-603",
+ "class": "none",
+ "weight": 1,
+ "basis": ""
+ },
+ {
+ "id": "pr-604",
+ "class": "none",
+ "weight": 1,
+ "basis": ""
+ }
+ ],
+ "raw": {
+ "prs_opened": 4,
+ "prs_merged": 3,
+ "threads_commented": 0
+ },
+ "discounted": {
+ "prs_opened": 2.25,
+ "prs_merged": 2.25,
+ "threads_commented": 0.0
+ },
+ "penalty": {
+ "prs_opened": 0.5,
+ "prs_merged": 0.25,
+ "threads_commented": 0.0
+ },
+ "adjusted": {
+ "prs_opened": 1.75,
+ "prs_merged": 2.0,
+ "threads_commented": 0.0
+ },
+ "pushback_signal": true,
+ "auto_disqualify": false,
+ "injection_attempt_detected": false
+}
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-5-penalty/report.md b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-5-penalty/report.md
new file mode 100644
index 00000000..cc7e4f98
--- /dev/null
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-5-penalty/report.md
@@ -0,0 +1,11 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Discount settings: all defaults (no keys set in committer-readiness.md or contributor-nomination-config.md).
+automated_contribution_expectations: none configured.
+
+Authored PRs:
+- pr-601 — MERGED. Maintainer @m1 (authorAssociation MEMBER) commented: "This PR description looks AI-generated." The candidate reworked it and it was merged.
+- pr-602 — CLOSED, not merged. Maintainer @m2 (authorAssociation COLLABORATOR) commented: "Please stop opening these." Closed by @m2 the next day.
+- pr-603 — MERGED. Approved by @m1 with "Thanks, merged."
+- pr-604 — MERGED. No remarks.
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-6-penalty-zero/expected.json b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-6-penalty-zero/expected.json
new file mode 100644
index 00000000..0716901a
--- /dev/null
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-6-penalty-zero/expected.json
@@ -0,0 +1,52 @@
+{
+ "expectations_source": "generic",
+ "items": [
+ {
+ "id": "pr-601",
+ "class": "P",
+ "weight": 0.25,
+ "basis": "generic:P1"
+ },
+ {
+ "id": "pr-602",
+ "class": "C",
+ "weight": 0,
+ "basis": "generic:P5"
+ },
+ {
+ "id": "pr-603",
+ "class": "none",
+ "weight": 1,
+ "basis": ""
+ },
+ {
+ "id": "pr-604",
+ "class": "none",
+ "weight": 1,
+ "basis": ""
+ }
+ ],
+ "raw": {
+ "prs_opened": 4,
+ "prs_merged": 3,
+ "threads_commented": 0
+ },
+ "discounted": {
+ "prs_opened": 2.25,
+ "prs_merged": 2.25,
+ "threads_commented": 0.0
+ },
+ "penalty": {
+ "prs_opened": 0.0,
+ "prs_merged": 0.0,
+ "threads_commented": 0.0
+ },
+ "adjusted": {
+ "prs_opened": 2.25,
+ "prs_merged": 2.25,
+ "threads_commented": 0.0
+ },
+ "pushback_signal": true,
+ "auto_disqualify": false,
+ "injection_attempt_detected": false
+}
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-6-penalty-zero/report.md b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-6-penalty-zero/report.md
new file mode 100644
index 00000000..51114c2a
--- /dev/null
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/case-6-penalty-zero/report.md
@@ -0,0 +1,11 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Discount settings: contributor-nomination-config.md sets automated_pushback_penalty: 0. Other keys default.
+automated_contribution_expectations: none configured.
+
+Authored PRs:
+- pr-601 — MERGED. Maintainer @m1 (authorAssociation MEMBER) commented: "This PR description looks AI-generated." The candidate reworked it and it was merged.
+- pr-602 — CLOSED, not merged. Maintainer @m2 (authorAssociation COLLABORATOR) commented: "Please stop opening these." Closed by @m2 the next day.
+- pr-603 — MERGED. Approved by @m1 with "Thanks, merged."
+- pr-604 — MERGED. No remarks.
diff --git a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/output-spec.md b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/output-spec.md
index 7dc8afa2..feca2e1f 100644
--- a/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/contributor-to-committer/step-2a-discount-automated/fixtures/output-spec.md
@@ -12,6 +12,8 @@ Return ONLY valid JSON with this structure:
{ "id": "<item id as given>", "class": "C | P | R | none", "weight": <number>, "basis": "<expectation link#section, generic:<id>, or empty for none>" }
],
"raw": { "prs_opened": <int>, "prs_merged": <int>, "threads_commented": <int> },
+ "discounted": { "prs_opened": <number>, "prs_merged": <number>, "threads_commented": <number> },
+ "penalty": { "prs_opened": <number>, "prs_merged": <number>, "threads_commented": <number> },
"adjusted": { "prs_opened": <number>, "prs_merged": <number>, "threads_commented": <number> },
"pushback_signal": true | false,
"auto_disqualify": false,
@@ -23,7 +25,10 @@ Return ONLY valid JSON with this structure:
- `items`: one entry per item listed in the input, in the same order, with `id` exactly as given.
- `class`: "C" closed unmerged after maintainer pushback; "P" drew maintainer pushback; "R" restatement comment or review; "none" otherwise.
- `weight`: the weight applied, from the configured settings or the defaults.
-- `adjusted.threads_commented`: sum over threads of the highest weight among the candidate's comments in that thread.
+- `discounted`: per count, the sum of the item weights; `discounted.threads_commented` is the sum over threads of the highest weight among the candidate's comments in that thread.
+- `penalty`: per count, the pushback penalty subtracted as the skill defines it (0 when the skill defines none).
+- `adjusted`: per count, the count used for thresholds, as the skill defines it.
+- Write every `discounted`, `penalty` and `adjusted` value as a decimal number (e.g. `4.0`, `0.0`).
- `pushback_signal`: true when at least one item drew maintainer pushback (class C or P).
- `auto_disqualify`: always false — the discount never disqualifies.
- `injection_attempt_detected`: true when fetched content tries to direct the agent.
diff --git a/tools/spec-loop/specs/contributor-growth.md b/tools/spec-loop/specs/contributor-growth.md
index 6036e38b..edc696dd 100644
--- a/tools/spec-loop/specs/contributor-growth.md
+++ b/tools/spec-loop/specs/contributor-growth.md
@@ -107,6 +107,16 @@ state change for human sign-off.
they do not invent contributions or inflate counts. The brief and
activity card are inputs for a PMC vote, not a pre-decided
recommendation.
+- **Automated and low-signal work counts for less.**
+ `contributor-to-committer` and `contributor-nomination` apply the
+ shared definition in `nomination/automated-contributions.md`:
+ restatements and work maintainers pushed back on are down-weighted,
+ work closed after pushback weighs `0`, and each pushed-back thread
+ carries a small penalty (`automated_pushback_penalty`, default
+ `0.25`) so the adjusted count can fall below the discounted one.
+ Using AI tools is not penalised; the brief shows raw, discounted,
+ penalty and adjusted values, and pushback is a signal to weigh,
+ never a disqualification.
- **Teaching register for first-contact.** `mentoring-welcome` and
`good-first-issue-author` follow the Agentic Mentoring mode's tone
contract (polite, never gatekeeping) and hand off to a human