You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "justinmclean (via GitHub)" <gi...@apache.org> on 2026/06/10 07:39:51 UTC
[PR] Fix setup-status eval prompts to surface decision rules [airflow-steward]
justinmclean opened a new pull request, #484:
URL: https://github.com/apache/airflow-steward/pull/484
## Summary
Add eval tests and discovered some bugs.
The step-1-command and step-3-adjust-decision evals extracted skill
sections that did not contain the rules for the fields under test, so
the model guessed instead of following the skill:
- step-1-command anchored on "Step 1 - Render the dashboard", which
never mentions the flags; repoint it at "Inputs" (the flag table)
and state in output-spec that no_adjust is independent of --format.
- step-3-adjust-decision could not see the command-mapping rules
(a peer "Step B" section) or the --no-adjust short-circuit (an intro
above the anchor). Nest the mapping under Step A, renumber the old
Step B/C, and add the short-circuit at the top of Step A.
Document the single-section extraction behaviour in the eval README
so future steps anchor step_heading at the section holding the rules.
## Type of change
<!-- Tick all that apply. -->
- [X] Skill change (`.claude/skills/<name>/`) — eval fixtures updated below
- [ ] Tool / bridge contract (`tools/<system>/*.md`)
- [ ] Python package (`tools/*/` with `pyproject.toml`)
- [ ] Groovy reference impl
- [ ] Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
- [ ] Documentation (`docs/`, `README.md`, `CONTRIBUTING.md`)
- [ ] Project template (`projects/_template/`)
- [ ] CI / dev loop (`prek`, workflows, validators)
- [ ] Other:
## Test plan
<!--
How you verified the change. Be specific. The reviewer reads this to
decide what to spot-check vs trust. Empty "ran the tests" doesn't help.
-->
- [X] `prek run --all-files` passes
- [ ] For Python packages touched: `uv run pytest` / `ruff check` / `mypy` passes
- [ ] For Groovy bridges touched: command-line invocation tested end-to-end
- [X] For skill changes: eval suite passes for the affected skill
(`PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/`)
- [ ] For skill *behaviour* changes: a new or updated eval fixture is included in this PR
(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
- [ ] Other:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix setup-status eval prompts to surface decision rules [airflow-steward]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #484:
URL: https://github.com/apache/airflow-steward/pull/484#issuecomment-4679069957
But I will wait for confirmation on that one @justinmclean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix setup-status eval prompts to surface decision rules [airflow-steward]
Posted by "justinmclean (via GitHub)" <gi...@apache.org>.
justinmclean commented on PR #484:
URL: https://github.com/apache/airflow-steward/pull/484#issuecomment-4678705499
Run teh review skill on this:
## Correctness
[blocking] step-2-present/fixtures/output-spec.md:17 — the output-spec, which is appended to the model's system prompt, literally states the answer: "Per the skill's OUTPUT CONTRACT, the correct answer is always verbatim." Cases 2 and 3 are supposed to test whether the model resists pressure to paraphrase; with the answer in-context they pass-stamp regardless of whether the skill's OUTPUT CONTRACT actually reached the model. Remove that line (the field-semantics lines above it are fine).
[advisory] step-1-command / step-2-present anchors verified correct — the OUTPUT CONTRACT blockquote and Step-A rules fall inside the extraction window, so the skill rule does reach the model. The leak above is the only wiring problem.
## Conventions
[blocking — verified] Malformed SPDX header in fixture report.md files. The header opens with <\!-- (literal backslash) instead of <!-- — confirmed in the raw bytes via cat -A. That is not a valid comment opener, so a license scanner keying on a proper comment may not recognise it. Affects ~13 fixture report.md files across all four steps. Likely a fixture-generation escaping bug; fix the generator/template, not each file by hand.
[advisory] evals/setup-status/README.md uses uv run --project tools/skill-evals — your convention is --directory (per your own runner notes). Will fail if copy-pasted.
Some more work needs to be done here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix setup-status eval prompts to surface decision rules [airflow-steward]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #484:
URL: https://github.com/apache/airflow-steward/pull/484#issuecomment-4680761408
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix setup-status eval prompts to surface decision rules [airflow-steward]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #484:
URL: https://github.com/apache/airflow-steward/pull/484
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org