You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by "justinmclean (via GitHub)" <gi...@apache.org> on 2026/07/28 23:57:54 UTC
[I] Correct the reviewer-routing spec claim about what the non-ASF eval suite asserts [magpie]
justinmclean opened a new issue, #999:
URL: https://github.com/apache/magpie/issues/999
## Summary
`tools/spec-loop/specs/reviewer-routing.md` says the non-ASF smoke suite asserts that
"area-match routing and load-aware fallback both work". The suite does not assert the second
half. Correct the sentence to describe what the fixtures actually check.
## Background
The specs under `tools/spec-loop/specs/` are the framework's source of truth, and this sentence
replaced a Known Gap — so it reads as proven coverage and will be trusted by anyone deciding
what still needs testing.
The suite has two cases. `case-1-non-asf-roster-area-match` does test area-match routing.
`case-2-non-asf-no-area-match` does **not** test a load-aware fallback: its `expected.json`
asserts `has_null_primary: true`, `no_eligible_reviewer: true`, and
`load_signal_shown: false` — no reviewer is selected and no load signal is shown. So the
sentence claims coverage that does not exist.
Note this issue is only about making the sentence match the fixtures. There is a separate,
larger question about *which* behaviour is correct for a no-area-match (the skill's own text
supports both readings) — that is a maintainer decision and explicitly out of scope here. Do
not change the fixture to match the sentence; change the sentence to match the fixture.
## Where to look
- `tools/spec-loop/specs/reviewer-routing.md` around line 120-124 — the Known-gap replacement
paragraph beginning "Non-ASF roster shape is now exercised".
- `tools/skill-evals/evals/non-asf-profile-smoke/step-reviewer-routing/fixtures/case-2-non-asf-no-area-match/expected.json`
— what case 2 actually asserts.
- `tools/skill-evals/evals/non-asf-profile-smoke/README.md` — the suite's own table already
describes case 2 correctly ("no-area-match → NO ELIGIBLE REVIEWER"); useful as a model for
the wording.
## Acceptance criteria
- [ ] The spec sentence describes only what the two fixtures assert.
- [ ] The spec no longer claims the suite covers load-aware fallback.
- [ ] No file under `tools/skill-evals/` is modified.
- [ ] `prek run --all-files` passes.
## Estimated effort
~30 minutes for someone new to the codebase.
## Getting started
[[How to contribute](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)](https://github.com/apache/airflow-steward/blob/main/CONTRIBUTING.md#your-first-contribution)
This issue was drafted with the help of an AI-assisted tool and reviewed by a Magpie maintainer before posting. If anything here is unclear or looks wrong, say so on the issue: a real person is reading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Correct the reviewer-routing spec claim about what the non-ASF eval suite asserts [magpie]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #999: Correct the reviewer-routing spec claim about what the non-ASF eval suite asserts
URL: https://github.com/apache/magpie/issues/999
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org