You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/09/24 10:12:50 UTC
[PR] feat(adversarial-review): other models' adversarial review, as a tool and substrate plugin [magpie]
potiuk opened a new pull request, #1366:
URL: https://github.com/apache/magpie/pull/1366
## Summary
- Adds `tools/adversarial-review`, a stdlib-only CLI. It runs other models' CLIs (Codex, Copilot, Gemini, Claude) read-only over a change and prints their findings as one advisory JSON report, de-duplicated across reviewers. `detect` reports which CLIs are installed and which harness is running (that one is skipped). `run` reviews a branch, a PR or a diff file.
- Publishes the tool as the `magpie-adversarial-review` substrate plugin, the same pattern as `vetted-ops`. The tool runs outside the sandbox, because the reviewer CLIs need network access and their own credentials, so it must live where the calling agent cannot rewrite it.
- Includes the design ([`docs/designs/2026-09-23-adversarial-review.md`](docs/designs/2026-09-23-adversarial-review.md)) and its implementation plan. This is PR 1 of 4. Still to come: `setup` detection and per-harness commands, a shared pre-PR block in every skill that runs `gh pr create`, and `with-reviewers:` in `pr-management-code-review`.
Why: today a second model's review only happens when a maintainer remembers to type one slash command, only for one reviewer, and only in Claude Code. The PRs Magpie opens itself get no second read.
## Type of change
- [x] Python package (`tools/*/` with `pyproject.toml`)
- [x] Documentation (`docs/`, `README.md`, `CONTRIBUTING.md`)
- [x] CI / dev loop (`prek`, workflows, validators)
- [x] Other: new substrate plugin, and a new `substrate:review` tool capability in the taxonomy
## Test plan
- [x] `prek run --all-files` passes
- [x] For Python packages touched: `pytest` (120 tests), `ruff check` and `mypy` pass for `tools/adversarial-review`; the `tools/dev` and `tools/skill-and-tool-validator` suites pass
- [x] Other:
- All reviewer CLIs are stubbed on a fake `PATH`; no test makes a model call.
- Every backend's command line is pinned by a snapshot. A test rejects known write-granting flags and any prompt text in argv.
- The privacy boundary is pinned: the input builder and `run` accept no parameter beyond the diff, the file list and the PR title and body. A tracker-shaped context in the environment and working tree never reaches the prompt.
- The runner is tested for:
- parallel runs;
- a timeout for one reviewer that does not block the others;
- a finished reply kept when a helper process lingers;
- a helper that calls `setsid` not stretching the timeout;
- termination on signal.
- `adversarial-review detect` was smoke-tested through the generated plugin path.
## RFC-AI-0004 compliance
- [x] **HITL**: findings are advisory. The tool exits 0 whenever it completes and never acts on a finding; the human decides.
- [x] **Sandbox**: the tool runs reviewer CLIs in their own read-only modes. MCP servers are switched off for `codex` and `claude`. It writes nothing to the repository. The sandbox exclusion itself comes in PR 2.
- [x] **Vendor neutrality**: four backends, one adapter each. The harness running the tool is detected and excluded.
- [x] **Privacy LLM**: reviewers see only content the PR will publish, by construction. The design explains why no privacy-llm gate applies, and states the residual risk: `codex` read-only mode does not confine reads, and `copilot`/`gemini` keep their MCP servers.
## Linked issues
None.
## Notes for reviewers (optional)
- **New capability value:** `substrate:review` is new. None of the existing substrate capabilities fit a second-model reviewer. Happy to fold it into an existing one if preferred.
- **Self-detection markers:** `CLAUDECODE=1` and `GEMINI_CLI=1` are confirmed. The Codex (`CODEX_SANDBOX`/`CODEX_THREAD_ID`) and Copilot (`COPILOT_CLI`) markers still need confirming from inside each CLI. `--self` overrides a wrong detection.
- **Command form:** the documented form is `uvx --from <plugin>/tools/adversarial-review adversarial-review …`, not `uv run --project …`, so that nothing is written into the read-only plugin cache.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(adversarial-review): other models' adversarial review, as a tool and substrate plugin [magpie]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #1366:
URL: https://github.com/apache/magpie/pull/1366#issuecomment-5813295176
Superseded by #1368, opened from a branch in `apache/magpie` so the rest of the stack can target it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(adversarial-review): other models' adversarial review, as a tool and substrate plugin [magpie]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #1366: feat(adversarial-review): other models' adversarial review, as a tool and substrate plugin
URL: https://github.com/apache/magpie/pull/1366
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org