You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/05/20 23:39:00 UTC
(tooling-agents) branch main updated (d27b301 -> ca4fa66)
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
from d27b301 ASVS L1 audit: consolidated report (redacted) [source: apache/mina @ accfb47]
new a8f95da read_and_triage v2.2: prior-comment ingestion, skip explanations, label fixes, fast-path, pagination, quality refinements
new 032d1ec asvs_orchestrate: renumber Top N Risks list and update heading after critical redaction
new ca4fa66 asvs_relevance_filter: forbid defense-in-depth escape hatch in DOWNGRADE
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
ASVS/agents/asvs_orchestrate.py | 85 ++++++++++++++++++++++++
ASVS/agents/asvs_relevance_filter.py | 115 +++++++++++++++++++++++++++++++-
gh-helper/README.md | 2 +-
gh-helper/agents/triage-and-pr.py | 123 +++++++++++++++++++++++++++++++++--
4 files changed, 317 insertions(+), 8 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
(tooling-agents) 02/03: asvs_orchestrate: renumber Top N Risks list and update heading after critical redaction
Posted by ak...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
commit 032d1eca82dfc2ff4695cd9cf6eb75d5ff416f16
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Wed May 20 16:22:14 2026 -0700
asvs_orchestrate: renumber Top N Risks list and update heading after critical redaction
- redact_consolidated already strips Critical risk items from the
Top N Risks numbered list via _strip_critical_list_items, but the
surviving items kept their original numbers (2, 3, 4, 5 with item
1 gone) and the heading still said Top 5 Risks even with 4 items
remaining; both visible on the May 20 mina L1 public report
- new step 3c.1 renumber pass in redact_consolidated: scan for any
heading matching Top N Risks at depth 2-6, walk the section until
the next heading, renumber surviving top-level list items
sequentially from 1, rewrite the heading count to match the
survivor count
- top-level only (no leading whitespace) so nested numbered
sub-bullets inside a risk description keep their original
numbering
- code-fence aware so example markdown inside fenced blocks is not
touched
- no-op when no Top N Risks section exists or all items were
redacted
- smoke-tested against mina shape (Top 5 with item 1 removed →
Top 4 with items 1-4), nested sub-bullets edge case, and
all-items-redacted edge case
---
ASVS/agents/asvs_orchestrate.py | 85 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 85 insertions(+)
diff --git a/ASVS/agents/asvs_orchestrate.py b/ASVS/agents/asvs_orchestrate.py
index 5e1a99f..ebe5219 100644
--- a/ASVS/agents/asvs_orchestrate.py
+++ b/ASVS/agents/asvs_orchestrate.py
@@ -713,6 +713,91 @@ async def run(input_dict, tools):
return '\n'.join(out)
redacted = _strip_critical_list_items(redacted)
+ # 3c.1. Renumber the Top N Risks numbered list and update
+ # its heading to match the surviving count. After 3c
+ # drops Critical risks (e.g. item 1 of a Top 5 list),
+ # the survivors keep their original numbers (2, 3, 4, 5)
+ # and the heading still says "Top 5 Risks" — both visible
+ # in the public report on the May 20 mina L1 run. This
+ # pass renumbers survivors sequentially from 1 and
+ # rewrites the heading to "Top {survivor_count} Risks".
+ #
+ # Scope: only the section under a heading matching
+ # `^#{2,6} Top N Risks`. List renumbering does not apply
+ # to numbered lists elsewhere in the document
+ # (remediation steps, acceptance criteria, etc.) —
+ # those should keep their original numbering even when
+ # individual items happen to mention Critical findings.
+ #
+ # Safety: code-fenced regions are skipped exactly as in
+ # 3a-3c. If no Top N Risks heading exists, this is a
+ # no-op.
+ def _renumber_top_risks(text):
+ lines = text.split('\n')
+ out = []
+ in_fence = False
+ top_heading_re = re.compile(
+ r'^(#{2,6})\s+Top\s+(\d+)\s+Risks?\s*$',
+ re.IGNORECASE,
+ )
+ boundary_re = re.compile(r'^#{1,6}(\s|$)')
+ # Only match top-level list items (no leading
+ # indent). Sub-bullets inside a risk's description
+ # — common pattern when a finding's narrative
+ # includes its own numbered breakdown — should
+ # keep their original numbering.
+ list_item_re = re.compile(r'^\d+\.\s')
+ i = 0
+ while i < len(lines):
+ line = lines[i]
+ stripped = line.lstrip()
+ if stripped.startswith('```'):
+ in_fence = not in_fence
+ out.append(line)
+ i += 1
+ continue
+ if in_fence:
+ out.append(line)
+ i += 1
+ continue
+ m = top_heading_re.match(line)
+ if not m:
+ out.append(line)
+ i += 1
+ continue
+ # Found Top N Risks heading. Park a placeholder
+ # in the output; rewrite it after counting the
+ # surviving items in the section below.
+ heading_prefix = m.group(1)
+ heading_idx = len(out)
+ out.append(line) # placeholder
+ i += 1
+ survivor_count = 0
+ while i < len(lines):
+ nxt = lines[i]
+ nxt_stripped = nxt.lstrip()
+ if nxt_stripped.startswith('```'):
+ in_fence = not in_fence
+ out.append(nxt)
+ i += 1
+ continue
+ if (not in_fence) and boundary_re.match(nxt_stripped):
+ break # next heading; stop scanning
+ if (not in_fence) and list_item_re.match(nxt):
+ survivor_count += 1
+ n = survivor_count # bind for closure
+ nxt = list_item_re.sub(
+ lambda mm, n=n: f"{n}. ",
+ nxt,
+ count=1,
+ )
+ out.append(nxt)
+ i += 1
+ if survivor_count > 0:
+ out[heading_idx] = f"{heading_prefix} Top {survivor_count} Risks"
+ return '\n'.join(out)
+ redacted = _renumber_top_risks(redacted)
+
sorted_ids = sorted(redacted_ids,
key=lambda s: int(re.search(r'(\d+)', s).group(1)) if re.search(r'(\d+)', s) else 0)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
(tooling-agents) 01/03: read_and_triage v2.2: prior-comment ingestion, skip explanations, label fixes, fast-path, pagination, quality refinements
Posted by ak...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
commit a8f95da7dd8db4498a67a8a3d258056f006a72ea
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Wed May 20 16:21:53 2026 -0700
read_and_triage v2.2: prior-comment ingestion, skip explanations, label fixes, fast-path, pagination, quality refinements
Prior comments on the issue now feed deep analysis as context. Skip
paths post explanation comments so users do not see a label without
any visible reasoning. Label default works correctly with the
gofannon UI submission semantics. A label-based fast-path skips
already-triaged issues without fetching their comments, and comment
fetching now paginates so the sentinel check sees the whole thread
on long-running issues. Citation grounding is stricter, inventory
parsing more robust, domain partitions consolidated when the model
overshoots the cap. Observability prints added throughout.
Prior discussion ingestion (main behavioral change):
- New _format_issue_comments helper renders non-bot, non-self
comments chronologically into a Prior discussion block in the
analysis prompt. Filters bots (user.type=Bot, login ending in
[bot]) and the agent own sentinel-tagged comments so re-runs do
not feed the model its own prior analyses.
- 30K total cap, 2.5K per comment, oldest-dropped-first when over
budget so the freshest discussion always survives.
- _ANALYSIS_SYSTEM gains a Prior Discussion Context section telling
the model to treat in-thread decisions (library choices, blockers,
external work filed) as more current than the issue body, cite
commenters by name, and lead with in-flight external work rather
than proposing duplicate internal work.
- Tested against ATR #1154 (SWHID): previous run proposed
reimplementing the algorithm in pure Python; new run correctly
identifies that @andrewmusselman is wrapping swhid-rs as asfswhid,
names the upstream blocker (swhid-rs#44 gitoxide merge), and
proposes no internal changes.
Skip-path commenting:
- New SENTINEL_SKIP and _build_skip_comment.
- pr_in_progress skips now post a brief comment naming the in-flight
PR(s) and explaining why no full triage was run.
- Idempotent via the new sentinel; re-runs check for SENTINEL_SKIP
before posting.
- Related-cluster info for the same issue is folded into the same
skip comment under a Related issues section rather than posted as
a second comment.
- Sentinel-skipped issues unchanged; the existing v2 triage comment
from the prior run is sufficient.
In-flight PR detection and diff suppression:
- Patch Discipline section in _ANALYSIS_SYSTEM forbids no-op
placeholders. When uncertain about a specific value (enum case,
constant, signature) the model must write a TODO comment and
route the question to open_questions instead of writing
self-assignments like x.field = x.field that look like working
code on first read.
- Same section instructs the model to set diffs to empty list when
prior discussion identifies a PR already implementing the work.
- Rendering-side belt-and-suspenders: _build_comment scans summary
and approach for PR #N references not in linked_prs (the
close-keyword-detected set) and suppresses Suggested patches in
favor of an In-flight work notice naming the PR(s).
Label default sentinel:
- Empty input from the gofannon UI now defaults to gh-helper
instead of disabling labeling. The previous in-input-dict check
was always true because gofannon submits declared input fields
even when blank.
- To explicitly disable labeling pass none, off, disabled, or -
(case-insensitive).
- ensure_repo_label and add_issue_label now log on every code path
(success, no-op, exception). Top-of-loop print confirms whether
labeling is enabled and which label name will be applied.
Label-based fast-path skip:
- After fetching open issues, partition into already-labeled and
unlabeled before the per-issue comment fetch. Labeled issues
skip straight to the result-row append with a distinguishing
summary; un-labeled issues continue through the existing
pipeline (including the comment-sentinel check, which now
functions as a safety net).
- Saves one paginated comments GET per labeled issue. On the ATR
triage corpus (103 open issues, 95 already labeled), that
removes 95 round trips per run, many of which would each be
multiple pages for long-thread issues.
- Prefiltered issues still appear in results with classification
skipped and a summary directing the user to strip the label or
set skip_already_triaged=false if they want to force re-triage.
They count in issues_skipped and issues_processed.
- No new input field. Tied to existing skip_already_triaged and
label inputs: fast-path fires when both are non-disabled.
Comment fetch pagination:
- New _fetch_all_comments helper follows GitHub Link header
rel=next pages, with per_page=100 up front to make most issues
single-page. Iterations capped at 50 defensively.
- Replaces the single gh_get(comments_url) at the sentinel-check
site. GitHub /issues/{N}/comments returns 30 by default and
caps at 100; on long-running issues like ATR #233 (40+ comments
across 8+ months), the bot prior triages lived on page 2+ and
were invisible to the sentinel scan, causing the agent to
re-triage on every run.
- Side benefit: prior-discussion ingestion downstream now sees
the full thread for long issues instead of just the first 30
comments, which materially improves analysis quality on
long-running threads.
Citation grounding refinements:
- snippet.rstrip on trailing whitespace before line splitting so
trailing newlines do not inflate computed line counts.
- Citations whose snippets contain no def / class / async def /
decorator anchor are dropped. Previous behavior was to fall back
to the longest line, which let prose explanations through as if
they were source citations.
- Per-issue print: Issue #N: Citation grounding: kept X, dropped Y.
Inventory parser robustness:
- Header regex accepts H2 through H4 (was H3 only).
- Strips markdown decorations from header text including bold,
inline code, and italic.
- Strips File: / Path: / FILE / PATH prefixes some prompts produce.
- Logs unresolved-header counts when model output does not match
any known file path.
Domain partition consolidation:
- Prompt strengthened with a STRICT REQUIREMENT to produce between
lo and hi domains total.
- Post-process: if the model overshoots the cap, keep the hi-1
largest partitions and merge the rest into a single shared bucket
with deduplicated files and concatenated concerns.
- Print: Domains: model produced X partitions, over the hi cap;
consolidated K smaller ones into shared.
Observability:
- Per-issue prints for citation grounding kept/dropped, in-flight
PR detection warnings, label apply success/failure.
- Per-run prints for inventory unresolved-header counts, domain
consolidation triggers, labeling-enabled banner, and label
fast-path count (N of M issues already carry the label,
processing remaining).
Tests at 43/43 (was 36). New: T37 blank-label-defaults, T38
PR-mention-suppresses-diffs, T39 normal-diffs-render-without-
mention, T40 pr_in_progress-posts-explanation, T41
pr_in_progress-skip-idempotent, T42 sentinel-on-page-2-detected,
T43 label-fast-path-skips-comment-fetch. Updated: T26 for the new
label sentinel semantics, T21 for the combined skip-plus-related
comment shape. FakeResp gained a headers field to support
pagination tests.
Per-call parameters dict (max_tokens 32768, reasoning_effort high,
temperature 1) left in for Opus deep-analysis as a fallback in
case the gofannon wrapper is fixed later. The wrapper currently
ignores per-call parameters and only honors UI-configured
Invokable Model parameters; gofannon-side issue filed separately.
---
gh-helper/README.md | 2 +-
gh-helper/agents/triage-and-pr.py | 123 ++++++++++++++++++++++++++++++++++++--
2 files changed, 120 insertions(+), 5 deletions(-)
diff --git a/gh-helper/README.md b/gh-helper/README.md
index 072558a..528614c 100644
--- a/gh-helper/README.md
+++ b/gh-helper/README.md
@@ -17,7 +17,7 @@ For each repo (cached per HEAD SHA, run once and shared across all issues):
Then for each issue:
-1. **Sentinel and PR-link checks** — skip issues already triaged by this agent, or skip with classification `pr_in_progress` if a PR is linked. For `pr_in_progress`, a brief explanatory comment is posted (idempotent via its own sentinel) noting which PR(s) the agent identified as in-flight, so the user isn't left wondering why the issue got a label without an explanation. Sentinel-skipped issues already have a full triage comment from the prior run, so no new comment is posted there.
+1. **Sentinel, label, and PR-link checks** — skip issues already triaged by this agent (label-based fast-path first, comment-sentinel scan as fallback), or skip with classification `pr_in_progress` if a PR is linked. The label fast-path filters out issues that already carry the configured label (default `gh-helper`) before fetching their comments, since labels are applied on every processing path — saves one paginated comments fetch per already-seen issue, which matters on long-thread is [...]
2. **Domain classification** — one Haiku call routes the issue to 1–3 application domains. Narrows the relevance-search to those domains.
3. **Relevance scoring** — Haiku, batched, scores files against the issue using *inventory entries* (not raw previews). Returns 0–10 per file. Top-K (8) selected with threshold ≥4, falling back to ≥2 if too few clear the bar.
4. **Staleness metrics** — computed locally from issue timestamps and comment activity. No LLM cost. Captures: days since created, days since last update, days since last comment, days since last *human* comment (skips bots and the agent's own comments), human comment count.
diff --git a/gh-helper/agents/triage-and-pr.py b/gh-helper/agents/triage-and-pr.py
index f493ab2..f26c533 100644
--- a/gh-helper/agents/triage-and-pr.py
+++ b/gh-helper/agents/triage-and-pr.py
@@ -2453,6 +2453,53 @@ If `triage_type` is `question`, `discussion`, `unrelated`, or `unclear`: set `cl
raise RuntimeError(f"GET {url} -> {r.status_code}: {r.text[:500]}")
return r
+ async def _fetch_all_comments(comments_url):
+ """Paginate through all comments on an issue. GitHub's
+ /issues/{N}/comments endpoint defaults to 30 per page and
+ caps at 100; long-running issues (#233 had 40+ comments
+ across 8+ months) push older content — including the bot's
+ prior triages — onto page 2+. Without pagination, the
+ sentinel-check sees only the first page and the agent
+ re-triages on every run.
+
+ We pull per_page=100 up front and then follow the Link
+ header for rel='next' until exhausted. Returns the combined
+ list of comment dicts in the same order GitHub returns them
+ (chronological, oldest first).
+ """
+ import re as _re_link
+ all_comments = []
+ # Tack per_page onto the base URL. comments_url comes back
+ # from GitHub without query string in the issues payload.
+ sep = "&" if "?" in comments_url else "?"
+ url = f"{comments_url}{sep}per_page=100"
+ # Bound iterations defensively so a misbehaving Link header
+ # can't loop us forever. 50 pages * 100 = 5000 comments
+ # which is comfortably beyond any real issue.
+ for _ in range(50):
+ if not url:
+ break
+ r = await gh_get(url)
+ all_comments.extend(r.json())
+ # Parse Link header for rel='next'. Format is
+ # <url1>; rel="next", <url2>; rel="last"
+ link_header = (
+ r.headers.get("link")
+ or r.headers.get("Link")
+ or ""
+ )
+ next_url = None
+ for part in link_header.split(","):
+ part = part.strip()
+ if 'rel="next"' not in part:
+ continue
+ m = _re_link.match(r"<([^>]+)>", part)
+ if m:
+ next_url = m.group(1)
+ break
+ url = next_url
+ return all_comments
+
async def gh_post(path, *, json_body=None):
url = path if path.startswith("http") else f"https://api.github.com{path}"
r = await http_client.post(url, headers=gh_headers, json=json_body)
@@ -2695,6 +2742,43 @@ If `triage_type` is `question`, `discussion`, `unrelated`, or `unclear`: set `cl
if max_issues:
issues = issues[:max_issues]
+ # ---------- 5.5. label-based fast-path filter ----------
+ # When labeling is enabled and skip_already_triaged is set
+ # (default), skip issues that already carry our label without
+ # bothering to fetch their comments for the sentinel check.
+ # The label is applied on every processing path (full triage,
+ # pr_in_progress skip, sentinel skip), so its presence is a
+ # reliable proxy for "we've already seen this issue". This
+ # saves one paginated GET per labeled issue — material on
+ # long-thread issues like ATR #233 that need multiple pages.
+ #
+ # The comment-sentinel check downstream still runs on the
+ # remaining (un-labeled) issues, as a safety net for the case
+ # where the label was manually stripped but the v2 sentinel is
+ # still in the comments.
+ prefiltered_labeled = []
+ if skip_already_triaged and label:
+ keep = []
+ for it in issues:
+ labels_on_issue = it.get("labels") or []
+ has_our_label = any(
+ (lbl.get("name") or "") == label
+ for lbl in labels_on_issue
+ )
+ if has_our_label:
+ prefiltered_labeled.append(it)
+ else:
+ keep.append(it)
+ if prefiltered_labeled:
+ print(
+ f"Label fast-path: {len(prefiltered_labeled)} of "
+ f"{len(issues)} issues already carry the '{label}' "
+ f"label; skipping comment-fetch on those. "
+ f"Processing {len(keep)} remaining.",
+ flush=True,
+ )
+ issues = keep
+
# ---------- 6. cross-issue context: open PRs + duplicate/related clusters ----------
# Both run once for the whole batch and feed every issue's
# decision-making in step 7. Cheap relative to the per-issue
@@ -2768,10 +2852,16 @@ If `triage_type` is `question`, `discussion`, `unrelated`, or `unclear`: set `cl
try:
# 7a. fetch existing comments once for sentinel checks
+ # AND for the prior-discussion ingestion downstream.
+ # Paginates because long-running issues exceed GitHub's
+ # 30-per-page default; without this, the bot's prior
+ # triages stay invisible to the sentinel check and the
+ # agent re-triages on every run.
existing_comments = []
if comments_url:
- cr = await gh_get(comments_url)
- existing_comments = cr.json()
+ existing_comments = await _fetch_all_comments(
+ comments_url
+ )
already_triaged = any(
SENTINEL_TRIAGE in (c.get("body") or "")
for c in existing_comments
@@ -3028,13 +3118,38 @@ If `triage_type` is `question`, `discussion`, `unrelated`, or `unclear`: set `cl
)
)
+ # Surface label-prefiltered issues in results and counts so
+ # the user can see them as skipped (with a distinguishing
+ # summary), rather than mysteriously vanishing from the
+ # output. Count them in issues_skipped and issues_processed,
+ # since they DID enter consideration — we just shortcut the
+ # comment fetch via the label fast-path.
+ for it in prefiltered_labeled:
+ results.append(
+ _result_row(
+ it.get("number"),
+ it.get("title") or "",
+ "skipped",
+ (
+ f"Already labeled '{label}'; skipping. "
+ f"Strip the label or set "
+ f"skip_already_triaged=false to force re-triage."
+ ),
+ [], "", "", False,
+ linked_prs=[],
+ related_issues=[],
+ )
+ )
+ skipped += len(prefiltered_labeled)
+ total_considered = len(issues) + len(prefiltered_labeled)
+
# Human-readable summary for gofannon's outputText slot. The
# structured fields are still returned alongside for programmatic
# consumption (e.g. by downstream agents reading via
# gofannon_client.call).
output_text = (
f"Triage of {repo} @ {branch}@{head_sha[:8]}\n"
- f"Issues processed: {len(issues)}\n"
+ f"Issues processed: {total_considered}\n"
f"Comments posted: {posted}\n"
f"Skipped: {skipped}\n"
f"Errors: {len(errors)}\n"
@@ -3046,7 +3161,7 @@ If `triage_type` is `question`, `discussion`, `unrelated`, or `unclear`: set `cl
return {
"outputText": output_text,
"repo": repo, "branch": branch, "head_sha": head_sha,
- "issues_processed": len(issues),
+ "issues_processed": total_considered,
"issues_commented": posted,
"issues_skipped": skipped,
"errors": errors,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
(tooling-agents) 03/03: asvs_relevance_filter: forbid defense-in-depth escape hatch in DOWNGRADE
Posted by ak...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
commit ca4fa6668f3c86cae919423b9a405d2d03a076e8
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Wed May 20 16:38:45 2026 -0700
asvs_relevance_filter: forbid defense-in-depth escape hatch in DOWNGRADE
- the filter prompt offered KEEP / DOWNGRADE / DROP; the DOWNGRADE
option was being used to resurrect findings whose root cause is in
a profile carve-out, keeping them alive at Low with a DiD framing
note. observed on logging-log4net F-009 (TelnetAppender bind
address downgraded from Low because profile delegates network
transport to deployment, but the lack of any configurable binding
address is a defense-in-depth gap) and F-010 (SMTP header CRLF
downgraded because configuration values are administrator-
controlled per trust boundary, however as defense-in-depth CRLF
sanitization would prevent edge-case header injection)
- prompt change tightens DOWNGRADE to root-cause-in-scope-but-
severity-overstated and explicitly forbids the escape hatch. adds
a Distinguishing DROP from DOWNGRADE paragraph that asks the
filter to consider whether the profile addresses the root cause
or only the severity, with domain-neutral examples (untrusted
data path the profile documents as not existing → DROP;
authenticated-access precondition that does not eliminate the
defect → DOWNGRADE). names the structural pattern (acknowledgment
of carve-out followed by contrastive resurrection) that must not
be used. points library-improvement-not-vulnerability findings at
DROP with reason feature request not vulnerability per profile
carve-out
- adds deterministic post-pass _is_did_escape_hatch that walks kept
findings after the LLM returns and moves any matching the pattern
from kept to dropped with confidence medium so they land in the
human review queue rather than the public report. detection
requires three conditions simultaneously: severity is Low or Info,
description contains a carve-out acknowledgment phrase from a
generic security-vocabulary list (trust boundary, out of scope,
delegated to, administrator-controlled, etc.), description
contains a defense-in-depth invocation phrase. stdout logs
per-section when the post-pass fires
- gate is intentionally Low/Info only since every observed instance
has landed there. broadening to Medium would risk false positives
on legitimate medium-severity defense-in-depth findings; revisit
if the pattern surfaces at Medium on a future run
- five-case smoke test passes: F-009 caught, F-010 caught, legit Low
without carve-out preserved, High with incidental DiD mention in
description preserved, Low with carve-out language but no DiD
framing preserved, hypothetical third-variant phrasing caught
---
ASVS/agents/asvs_relevance_filter.py | 115 ++++++++++++++++++++++++++++++++++-
1 file changed, 112 insertions(+), 3 deletions(-)
diff --git a/ASVS/agents/asvs_relevance_filter.py b/ASVS/agents/asvs_relevance_filter.py
index a358381..051884e 100644
--- a/ASVS/agents/asvs_relevance_filter.py
+++ b/ASVS/agents/asvs_relevance_filter.py
@@ -1078,6 +1078,55 @@ async def run(input_dict, tools):
print(f"[filter] {len(chapter_groups)} chapters → {len(batches)} batches")
+ # Safety net for the defense-in-depth escape hatch. The prompt
+ # forbids resurrecting a scope-carved-out finding as a Low
+ # "defense-in-depth gap"; this post-pass catches the pattern in
+ # case the LLM emits it anyway. Detection requires (a) the
+ # description acknowledges a profile carve-out applies AND
+ # (b) the description invokes defense-in-depth framing AND
+ # (c) the surviving severity is Low or Info. All three together
+ # are the exact escape-hatch shape we observed on the
+ # logging-log4net F-009 (Telnet bind) and F-010 (SMTP header)
+ # findings. Findings matching the pattern are moved from kept
+ # to dropped with confidence=medium so they land in the human
+ # review queue rather than being silently scrubbed.
+ def _is_did_escape_hatch(finding):
+ desc = (finding.get("description") or "").lower()
+ if not desc:
+ return False
+ sev = (finding.get("severity") or "").strip().lower()
+ if sev not in ("low", "info"):
+ return False
+ carve_phrases = (
+ "trust boundary",
+ "out of scope",
+ "delegated to",
+ "deployment concern",
+ "deployment manager",
+ "deployer's responsibility",
+ "deployer is responsible",
+ "administrator-controlled",
+ "administrator controlled",
+ "configuration-controlled",
+ "configuration controlled",
+ "dev-only",
+ "per profile",
+ "per the profile",
+ "downgraded from",
+ "documented design decision",
+ "profile delegates",
+ )
+ did_phrases = (
+ "defense-in-depth",
+ "defense in depth",
+ "as defense in",
+ "as a defense-in-depth",
+ "as a defense in depth",
+ "defense-in-depth gap",
+ "defense in depth gap",
+ )
+ return any(p in desc for p in carve_phrases) and any(p in desc for p in did_phrases)
+
FILTER_PROMPT_TEMPLATE = (
"You are a security audit triage reviewer.\n\n"
"You have the project's own Security Profile (synthesized from the "
@@ -1085,13 +1134,42 @@ async def run(input_dict, tools):
"reports. For each finding in each report, decide one of:\n\n"
"- **KEEP**: real, in-scope finding for this project's threat model. "
"Pass through unchanged.\n"
- "- **DOWNGRADE**: real finding but severity overstated given the "
- "project's threat model. Adjust severity and add a one-line note in "
- "the description.\n"
+ "- **DOWNGRADE**: real finding whose root cause is in scope but whose "
+ "severity is overstated. Adjust severity and add a one-line note in "
+ "the description. Use DOWNGRADE only when the underlying threat is "
+ "real for this project — not as a way to keep a scope-carved-out "
+ "finding alive at lower severity.\n"
"- **DROP**: explicitly out of scope per the profile (delegated to "
"deployment manager, dev-only component, documented design decision, "
"documentation already exists, ASVS category not applicable). Provide "
"a one-line reason that cites the profile section.\n\n"
+ "**Distinguishing DROP from DOWNGRADE**: ask whether the profile "
+ "addresses the finding's ROOT CAUSE (the data flow, the trust "
+ "assumption, the threat-model premise) or only its severity. If "
+ "the profile makes the threat premise moot — for example, the "
+ "finding's exploit requires an untrusted data path that the "
+ "profile documents as not existing, or a trust boundary the "
+ "profile documents as not crossed — that is DROP. If the profile "
+ "reduces but does not eliminate the practical impact — for "
+ "example, the exploit requires authenticated access or a "
+ "deployer-controlled precondition that does not eliminate the "
+ "underlying defect — that is DOWNGRADE.\n\n"
+ "**Defense-in-depth is not an escape hatch.** When a finding's "
+ "premise is carved out by the profile (no untrusted data path "
+ "exists, the surface is configuration-only, the component is "
+ "dev-only, the concern is delegated to the deployer), the correct "
+ "action is DROP. Recasting such a finding as a Low 'defense-in-"
+ "depth gap' is forbidden. If you find yourself writing a "
+ "downgrade note of the shape \"profile says X is out of scope, "
+ "BUT as defense-in-depth...\" or \"trust boundary excludes this, "
+ "HOWEVER...\" or \"DOWNGRADED to Low because [profile carve-out], "
+ "but...\" — stop and use DROP instead. The finding's own "
+ "acknowledgement of the carve-out is itself evidence the carve-out "
+ "applies; do not then resurrect the finding at lower severity. "
+ "Library-improvement suggestions that would be nice to have but "
+ "are not vulnerabilities under the project's threat model belong "
+ "in DROP with a reason like 'feature request, not vulnerability "
+ "per profile carve-out', not in DOWNGRADE.\n\n"
"For every DROP, also assign a CONFIDENCE level:\n"
"- **high**: the profile EXPLICITLY addresses this finding type (e.g. "
"the profile literally says 'TLS is deployment manager's responsibility' "
@@ -1252,6 +1330,37 @@ async def run(input_dict, tools):
dropped = rep.get("dropped_findings", []) or []
promoted = rep.get("promoted_positive_controls", []) or []
+ # Safety net: catch defense-in-depth escape-hatch downgrades the
+ # prompt rule missed. These get moved from kept to dropped with
+ # confidence=medium so a human reviewer can see them in the
+ # review queue and confirm the post-pass got it right.
+ escape_hatch_drops = []
+ survivors = []
+ for f in kept:
+ if _is_did_escape_hatch(f):
+ escape_hatch_drops.append(f)
+ else:
+ survivors.append(f)
+ if escape_hatch_drops:
+ kept = survivors
+ for f in escape_hatch_drops:
+ dropped.append({
+ "original_id": f.get("finding_id") or f.get("id") or "?",
+ "title": f.get("title", "?"),
+ "severity": f.get("severity", "?"),
+ "reason": (
+ "defense-in-depth escape hatch: finding "
+ "acknowledges a profile carve-out applies but was "
+ "kept at Low with DiD framing. Filter prompt rule "
+ "requires DROP in this case; post-pass corrected. "
+ "Queued for review."
+ ),
+ "confidence": "medium",
+ })
+ print(f"[filter] {key}: post-pass dropped "
+ f"{len(escape_hatch_drops)} defense-in-depth escape-hatch "
+ f"finding(s)")
+
total_kept += len(kept)
total_dropped += len(dropped)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org