You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@magpie.apache.org by "vaquarkhan (via GitHub)" <gi...@apache.org> on 2026/06/30 15:50:29 UTC

[I] Proposal: pr-integrity-gate skill deterministic code-quality gates on PR diffs [magpie]

vaquarkhan opened a new issue, #658:
URL: https://github.com/apache/magpie/issues/658

   ### What should happen
   
   ## Title
   
   Add `pr-integrity-gate` skill: deterministic, air-gapped code-quality gates (density, design, dependency, invariant) on PR diffs
   
   ## Summary
   
   Magpie's PR management suite excels at **agent-judgment review** (code review, slop detection, mentoring). What it currently lacks is a **quantitative, deterministic, no-LLM quality gate** that runs on the PR diff before the agent-driven review engages - catching scaffolding-heavy code, design-rule violations, phantom imports, and invariant failures with reproducible, air-gapped logic that needs no API key.
   
   I propose a new skill in the `repo-health` family that fills this gap by integrating deterministic gates from an existing, published, Apache-2.0 prototype: [**AIV Integrity Gate**](https://github.com/vaquarkhan/aiv-integrity-gate) (single shaded JAR on Maven Central, diff-scoped, air-gapped, supports Java/Python/Go/Rust/JS/TS/C/C++/Ruby/Shell).
   
   ## Problem
   
   | What Magpie has today | What's missing |
   |--|--|
   | `slop-detection` (structural: metadata signals - fork floods, area sprawl, ticket-style titles) | No **per-file quantitative density** check (logic-density ratio, entropy) to catch boilerplate/scaffolding code that slop-detection's structural heuristics miss |
   | `pr-management-code-review` (agent-judgment line-by-line review) | No **declarative design-rule enforcement** — a project cannot say "forbid `System.exit`" or "require `ExpireSnapshots` when `expireSnapshots` keyword appears" without relying on agent judgment |
   | `dependency-audit` (CVE check on declared deps) | No **import-vs-lockfile validation** on the PR diff — typosquatting/phantom-dependency vector unchecked |
   | No doc-health gate | No validation of broken paths, cross-references, or fabricated commands in changed documentation |
   
   These are **complementary** to agent-judgment, not competing — they run before the agent review and filter out PRs that fail deterministic bars, saving agent tokens and reviewer time.
   
   ## Proposed solution: `pr-integrity-gate` skill
   
   A new skill (family: `repo-health`, mode: `Triage`) that:
   
   1. Fetches the PR diff via Magpie's existing `tools/github/` adapter.
   2. Runs four deterministic gates on the changed files — **no LLM call, no network (air-gapped)**:
      - **Density gate** - logic-density ratio and Shannon entropy per file; flags low-signal scaffolding.
      - **Design gate** - YAML-defined forbidden/required patterns (supports AI-slop markers: emoji, "Generated by AI", attribution patterns).
      - **Dependency gate** -validates new imports in the diff match the project's lockfile (pom.xml, requirements.txt, package.json, go.mod).
      - **Invariant gate** - merge-conflict markers, TBD/FIXME/XXX placeholders.
      - *(Optional)* **Doc-integrity gate** — broken paths, cross-refs, fabricated commands in changed .md/.rst/.txt.
   3. Surfaces findings as a structured report (pass/fail per gate, per-file scores).
   4. Proposes an action to the maintainer (comment findings on PR / pass to full review / skip).
   5. Follows Magpie's HITL contract: **propose → confirm → apply** (never auto-posts).
   
   ### How it runs
   
   The skill invokes a single shaded JAR (`aiv-cli.jar` from Maven Central) on the local diff — same execution model as Groovy tools already in the repo. No daemon, no external service, no LLM tokens consumed. The JAR's exit code (0=pass, 1=fail) drives the skill's disposition logic.
   
   Alternatively, the gate logic can be reimplemented as a Python tool under `tools/pr-integrity-gate/` if the PMC prefers pure-Python over a JVM dependency.
   
   ### Configuration (adopter-side)
   
   The adopter commits `.aiv/config.yaml` and `.aiv/design-rules.yaml` in their repo (or under `.apache-magpie-overrides/pr-integrity-gate/`). Example:
   
   ```yaml
   # .aiv/config.yaml
   schema_version: 1
   gates:
     - id: density
       enabled: true
       config:
         ldr_threshold: 0.25
         entropy_threshold: 4.0
     - id: design
       enabled: true
       config:
         rules_path: .aiv/design-rules.yaml
     - id: dependency
       enabled: true
     - id: invariant
       enabled: true
   ```
   
   ## How this complements existing skills
   
   ```
   PR opened
     │
     ├─▶ slop-detection (structural metadata signals) ──▶ early-exit if crystal-clear slop
     │
     ├─▶ pr-integrity-gate (deterministic code gates) ──▶ fail-fast if density/design/dep/invariant violated
     │
     └─▶ pr-management-code-review (agent-judgment)  ──▶ deep line-by-line review on remaining PRs
   ```
   
   The three layers are additive: structural → quantitative → judgment. Each filters progressively, so the expensive agent review only runs on PRs that clear both deterministic bars.
   
   ## Compatibility with Magpie's principles (RFC-AI-0004)
   
   | Principle | How `pr-integrity-gate` complies |
   |--|--|
   | HITL on every state change | Propose → confirm → apply; never auto-posts |
   | Secure sandbox | Single JAR invocation, no network, no secrets |
   | Vendor neutrality | No LLM involved; pure deterministic logic |
   | Privacy by design | Air-gapped; code never leaves the machine |
   | Write-access discipline | Findings are drafted for review, not posted autonomously |
   
   ## Prototype
   
   A working, published implementation exists: **[AIV Integrity Gate](https://github.com/vaquarkhan/aiv-integrity-gate)**
   
   - Apache-2.0 licensed, `io.github.vaquarkhan` on Maven Central (v1.0.4)
   - Single shaded uber JAR (`aiv-cli-1.0.4.jar`), ~4 MB
   - Diff-scoped by default (compares PR branch to base)
   - Supports Java, Python, Go, Rust, Kotlin, Scala, JS, TS, C, C++, Ruby, Shell
   - Structured output: JSON (`--output-json`), SARIF 2.1.0 (`--output-sarif`), GitHub Checks (`--publish-github-checks`)
   - GitHub Actions composite action: `vaquarkhan/aiv-integrity-gate@v1`
   - 100% line coverage enforced per module
   - 5 gates shipped: density, design, dependency, invariant, doc-integrity
   - CI-validated on this repository itself (dogfooding)
   
   The prototype is ready for integration as-is (invoke the JAR) or as a reference for a pure-Python reimplementation.
   
   ## Implementation plan
   
   1. **Issue (this):** confirm interest and direction with the PMC.
   2. **Small first PR:** add "S6: low logic density" as a soft signal in `slop-detection.md` — a one-paragraph markdown edit that surfaces the density concept without any new tooling. Gets early visibility.
   3. **Main PR (`pr-integrity-gate` skill):** full SKILL.md + step-detail files + eval fixtures, following the `write-skill` scaffold. Invokes `aiv-cli.jar` (or a Python port) for the gate execution.
   4. **Follow-up:** wire `pr-integrity-gate` as a pre-step in `pr-management-code-review` (opt-in via adopter override).
   
   ## Anti-goals
   
   - Not replacing `pr-management-code-review` (agent review is complementary, not competing).
   - Not replacing `slop-detection` (structural metadata signals remain independently valuable).
   - Not requiring JVM on the adopter's machine if the PMC prefers a Python port.
   - Not sending code to any external service — the entire point is air-gapped, deterministic, reproducible.
   
   
   
   ### Why
   
   Magpie's PR management pipeline today relies on two layers: **structural metadata heuristics** (slop-detection: fork floods, area sprawl, ticket-style titles) and **agent-judgment** (pr-management-code-review: LLM-driven line-by-line analysis). Both are valuable, but there is no **quantitative, deterministic, no-LLM code-quality gate** between them.
   
   The concrete gaps:
   
   1. **No per-file density metric.** A PR can pass slop-detection (metadata looks fine) yet consist of 80% empty scaffolding and boilerplate. The agent review eventually catches this, but it burns tokens and reviewer time on something a simple logic-density ratio + entropy calculation can flag in milliseconds.
   
   2. **No declarative design-rule enforcement.** A project cannot express "forbid `System.exit` in diffs" or "require `ExpireSnapshots` when `expireSnapshots` appears" as a checked invariant — only as agent judgment that may or may not fire.
   
   3. **No import-vs-lockfile validation on the diff.** `dependency-audit` checks declared deps for known CVEs. It does NOT check whether new `import` statements in the PR actually correspond to a declared dependency — leaving the typosquatting / phantom-dependency supply-chain vector unchecked.
   
   4. **No doc-integrity gate.** Changed documentation can introduce broken paths, fabricated commands, or dangling cross-references with no automated catch.
   
   This violates the spirit of RFC-AI-0004 §2 (secure sandbox / defense in depth) — deterministic pre-filtering before agent-judgment is a defense-in-depth layer for code quality, just as the sandbox is for execution security. It also aligns with §3 (vendor neutrality): a gate that needs no LLM is the most vendor-neutral possible.
   
   A working prototype exists: **[AIV Integrity Gate](https://github.com/vaquarkhan/aiv-integrity-gate)** — Apache-2.0, published to Maven Central (`io.github.vaquarkhan:aiv-cli:1.0.4`), single shaded JAR, diff-scoped, air-gapped, 5 gates (density, design, dependency, invariant, doc-integrity), supports Java/Python/Go/Rust/JS/TS/C/C++/Ruby/Shell, 100% line coverage, GitHub Actions composite action, JSON + SARIF output.
   
   ### Which layer
   
   Skill (`.claude/skills/<name>/`)
   
   ### Boundary conditions (optional)
   
   _No response_
   
   ### Out of scope (optional)
   
   ## Out of scope
   
   - **Not replacing `pr-management-code-review`** — agent-judgment review is complementary; this skill is a pre-filter, not a substitute.
   - **Not replacing `slop-detection`** — structural metadata signals remain independently valuable and trigger before this skill.
   - **Not proposing Agentic Autonomous (auto-close/merge)** based on gate results — findings are always surfaced to the maintainer for decision.
   - **Not requiring JVM on every adopter's machine** — if the PMC prefers, the gate logic can be reimplemented as a Python tool (no JAR). The prototype is the reference, not the mandated runtime.
   - **Not adding new write-access patterns** — the skill uses the same `gh pr comment` / propose-confirm-apply flow as `pr-management-code-review`.
   - **Not touching security-issue handling, CVE flows, or private content** — this skill operates on public PR diffs only.
   
   ### References (optional)
   
   ## References
   
   - **Prototype (working, published):** https://github.com/vaquarkhan/aiv-integrity-gate
     - Maven Central: `io.github.vaquarkhan:aiv-cli:1.0.4`
     - GitHub Action: `vaquarkhan/aiv-integrity-gate@v1`
     - 5 gates: density, design, dependency, invariant, doc-integrity
     - Structured output: `--output-json` (schema v2), `--output-sarif` (SARIF 2.1.0), `--publish-github-checks`
     - Languages: Java, Python, Go, Rust, Kotlin, Scala, JS, TS, C, C++, Ruby, Shell
   - **RFC-AI-0004** (Magpie normative principles) - §2 (sandbox/defense-in-depth), §3 (vendor neutrality), §6 (privacy by design)
   - **Existing complementary skills:**
     - `skills/pr-management-code-review/slop-detection.md` - structural metadata signals (H1–H5, S1–S5); this proposal adds quantitative code signals
     - `skills/dependency-audit/SKILL.md` -CVE audit on declared deps; this proposal adds import-vs-lockfile on the diff
     - `skills/license-compliance-audit/SKILL.md` - license/NOTICE compliance (complementary read-only audit pattern)
   - **Also maintained by the author:**
     - https://github.com/vaquarkhan/compliance-agent-skills - 30 deterministic compliance Agent Skills (HIPAA, PCI-DSS, SOC 2, etc.) with Presidio PHI redaction
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@magpie.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org