You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "justinmclean (via GitHub)" <gi...@apache.org> on 2026/06/05 16:25:51 UTC
[PR] PR 454 evals tests [airflow-steward]
justinmclean opened a new pull request, #459:
URL: https://github.com/apache/airflow-steward/pull/459
## Summary
Added eval tests for PR 454. Not that two tests currently fail due to issues with the skill.
PASS step-2.5-slop-detection/case-1-crystal-clear-slop
PASS step-2.5-slop-detection/case-2-one-hard-three-soft
PASS step-2.5-slop-detection/case-3-one-hard-two-soft-note
PASS step-2.5-slop-detection/case-4-two-soft-note
PASS step-2.5-slop-detection/case-5-genuine-silent
PASS step-2.5-slop-detection/case-6-prompt-injection
FAIL step-2.5-slop-detection/case-7-legit-team-fork-false-positive
$.outcome: expected='note-only', actual='early-exit'
PASS step-2.5-slop-detection/case-8-real-pr-352-rename
FAIL step-2.5-slop-detection/case-9-h1-undetectable-from-real-payload
$.fired.hard: length mismatch (actual=1, expected=2)
$.outcome: expected='early-exit', actual='note-only'
## Type of change
<!-- Tick all that apply. -->
- [ ] Skill change (`.claude/skills/<name>/`) — eval fixtures updated below
- [ ] Tool / bridge contract (`tools/<system>/*.md`)
- [ ] Python package (`tools/*/` with `pyproject.toml`)
- [ ] Groovy reference impl
- [ ] Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
- [ ] Documentation (`docs/`, `README.md`, `CONTRIBUTING.md`)
- [ ] Project template (`projects/_template/`)
- [ ] CI / dev loop (`prek`, workflows, validators)
- [X] Other: added tests
## Test plan
- [X] `prek run --all-files` passes
- [ ] For Python packages touched: `uv run pytest` / `ruff check` / `mypy` passes
- [ ] For Groovy bridges touched: command-line invocation tested end-to-end
- [X] For skill changes: eval suite passes for the affected skill
(`PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/`)
- [ ] For skill *behaviour* changes: a new or updated eval fixture is included in this PR
(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
- [ ] Other:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PR 454 evals tests [airflow-steward]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #459:
URL: https://github.com/apache/airflow-steward/pull/459#issuecomment-4637640888
Thanks! I cherry-picked your commits into my branch so they could run altogether :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PR 454 evals tests [airflow-steward]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed pull request #459: PR 454 evals tests
URL: https://github.com/apache/airflow-steward/pull/459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] PR 454 evals tests [airflow-steward]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #459:
URL: https://github.com/apache/airflow-steward/pull/459#issuecomment-4637715706
Merged as part of https://github.com/apache/airflow-steward/pull/454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org