You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/06/03 08:33:56 UTC
(tooling-agents) branch main updated: asvs: tighten Informational severity, generalize scope check, cite files for positive patterns
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
The following commit(s) were added to refs/heads/main by this push:
new 946b4a6 asvs: tighten Informational severity, generalize scope check, cite files for positive patterns
946b4a6 is described below
commit 946b4a69f7bb31501a003217d61b0c7f5513c147
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Wed Jun 3 01:33:42 2026 -0700
asvs: tighten Informational severity, generalize scope check, cite files for positive patterns
* Scope Check (bundle, audit): replaced enumerated ASVS examples
(7.2.3, V10.4.x, 4.4.x, 1.5.1, V5.2.x) with a generic three-question
evaluation framework plus an anti-stretching warning. Models
over-index on enumerated cases; principles generalize. Absences now
route to N/A coverage rather than generating Informational findings.
* Informational definition (bundle): restricted to the downgrade-from-Low
case -- a real, concrete code defect with file:line references but no
clear attack scenario. Findings whose body would contain 'not
applicable', 'feature absent', 'delegated to', or 'no X in this
codebase' belong in N/A coverage, not findings.
* Filter safety net (relevance_filter): added _is_informational_absence
next to _is_did_escape_hatch. Drops Info findings matching absence
phrases, no-X-in-Y regex, or missing file references. High-confidence
drops, routed to drop log rather than review queue.
* Positive pattern file refs (bundle, audit): output format now requires
a file:line citation per positive pattern. The consolidator's
Supporting Files column populates from these references; previously
sparse because the prompts never required them.
Validated on extensions-cli rerun: 12 Informational -> 0, Low
calibration unchanged, review queue empty, supporting-files column
now sources from the audit output.
---
ASVS/agents/asvs_audit.py | 54 ++++++++----------
ASVS/agents/asvs_bundle.py | 63 +++++++++++----------
ASVS/agents/asvs_relevance_filter.py | 103 +++++++++++++++++++++++++++++++++++
3 files changed, 160 insertions(+), 60 deletions(-)
diff --git a/ASVS/agents/asvs_audit.py b/ASVS/agents/asvs_audit.py
index 7780061..3026ec8 100644
--- a/ASVS/agents/asvs_audit.py
+++ b/ASVS/agents/asvs_audit.py
@@ -902,36 +902,28 @@ Follow ALL of these analysis requirements:
### Scope Check — do this FIRST, before generating any findings
-Some ASVS requirements target SPECIFIC architectural patterns and apply
-only to systems exhibiting those patterns. Before generating findings,
-verify the audited codebase actually uses the pattern this requirement
-targets. If not, produce a single N/A finding explaining why, and STOP.
-
-Concrete examples (not exhaustive):
-- **Requirements about REFERENCE tokens** (e.g. 7.2.3 "Reference tokens
- unique with 128 bits entropy") apply only to systems issuing opaque
- reference tokens that are looked up server-side for state. Systems
- using self-contained tokens (JWT, signed cookies, PASETO) are OUT OF
- SCOPE — their security is governed by signing/algorithm requirements
- in V9, not entropy of reference identifiers. DO NOT apply
- reference-token requirements to JWT signing keys.
-- **OAuth Authorization Server requirements** (e.g. V10.4.x) apply only
- to systems acting as an OAuth provider issuing tokens to third-party
- clients. Systems that only CONSUME OAuth (OAuth clients) are out of
- scope for these requirements.
-- **WebSocket requirements** (e.g. 4.4.x) apply only to systems using
- the WebSocket protocol.
-- **XML parser requirements** (e.g. 1.5.1) apply only when the codebase
- parses XML.
-- **File-upload requirements** (V5.2.x) apply only when the codebase
- accepts and processes user-uploaded files.
-
-If the audited codebase does not exhibit the architectural pattern this
-requirement targets, return a single N/A finding with a clear
-explanation of WHY this requirement is not in scope (e.g. "Codebase uses
-self-contained JWTs validated cryptographically; this requirement
-applies only to opaque reference tokens"). Do NOT stretch the
-requirement to fit some loosely related aspect of the code.
+Each ASVS requirement is written assuming the audited code implements a
+specific technology, protocol, component role, feature, or data type.
+A requirement applies only when the audited code actually meets that
+architectural assumption.
+
+For each requirement, before generating any finding:
+1. Identify the architectural assumption the requirement embeds — what
+ protocol, technology, component role, feature, or data type the
+ requirement governs.
+2. Verify the audited codebase actually exhibits that assumption.
+3. If the assumption is not met, mark the requirement as N/A in the
+ coverage table with a one-line reason. Do NOT generate a finding for
+ it — absences are not findings, and findings exist only when there
+ is concrete code to describe a defect against.
+
+Do not stretch a requirement to fit thematically similar but
+architecturally different code. If the audited codebase implements
+something adjacent to the requirement's target — a different token
+format, a different transport, a different protocol role, a different
+component type — the requirement is N/A for the parts of the standard
+that target the specific pattern, even when other requirements in the
+same control family do apply.
### Core Principle: Existence ≠ Application
For each security control found:
@@ -1238,7 +1230,7 @@ Required structure:
- Location with file, function, line numbers
- Related functions checked
- Description, vulnerable code, data flow, PoC, impact, remediation
-5. **Positive Security Patterns** — what's done well
+5. **Positive Security Patterns** — what's done well, each with a `file:line` reference to the specific code
6. **Architecture Observations**
7. **Recommendations Summary** — Immediate/Short-term/Long-term
8. **Appendix: Files Analyzed** — collapsible list
diff --git a/ASVS/agents/asvs_bundle.py b/ASVS/agents/asvs_bundle.py
index da63226..8f46b95 100644
--- a/ASVS/agents/asvs_bundle.py
+++ b/ASVS/agents/asvs_bundle.py
@@ -1042,30 +1042,28 @@ Follow ALL of these analysis requirements:
### Scope Check — do this FIRST for EACH requirement, before generating findings
-Bundle audits cover multiple ASVS requirements simultaneously. Some
-target SPECIFIC architectural patterns and apply only to systems that
-exhibit those patterns. Per requirement: verify the audited codebase
-actually uses the targeted pattern. If not, produce a single N/A
-finding for that requirement explaining why, and move on.
-
-Concrete examples (not exhaustive):
-- **Requirements about REFERENCE tokens** (e.g. 7.2.3) apply only to
- systems issuing opaque reference tokens looked up server-side.
- Systems using self-contained tokens (JWT, signed cookies, PASETO)
- are OUT OF SCOPE — their security is governed by signing/algorithm
- requirements in V9. DO NOT apply reference-token requirements to
- JWT signing keys.
-- **OAuth Authorization Server requirements** (V10.4.x) apply only to
- systems acting as an OAuth provider. OAuth clients are out of scope.
-- **WebSocket requirements** (4.4.x) apply only when WebSocket is used.
-- **XML parser requirements** (1.5.1) apply only when XML is parsed.
-- **File-upload requirements** (V5.2.x) apply only when user file
- uploads are accepted.
-
-If the codebase does not exhibit the targeted pattern for a given
-requirement, return a single N/A finding for THAT requirement (not all)
-with a clear explanation of WHY it is not in scope. Do NOT stretch a
-requirement to fit some loosely related aspect of the code.
+Each ASVS requirement is written assuming the audited code implements a
+specific technology, protocol, component role, feature, or data type.
+A requirement applies only when the audited code actually meets that
+architectural assumption.
+
+For each requirement, before generating any finding:
+1. Identify the architectural assumption the requirement embeds — what
+ protocol, technology, component role, feature, or data type the
+ requirement governs.
+2. Verify the audited codebase actually exhibits that assumption.
+3. If the assumption is not met, mark the requirement as N/A in the
+ coverage table with a one-line reason. Do NOT generate a finding for
+ it — absences are not findings, and findings exist only when there
+ is concrete code to describe a defect against.
+
+Do not stretch a requirement to fit thematically similar but
+architecturally different code. If the audited codebase implements
+something adjacent to the requirement's target — a different token
+format, a different transport, a different protocol role, a different
+component type — the requirement is N/A for the parts of the standard
+that target the specific pattern, even when other requirements in the
+same control family do apply.
### Core Principle: Existence ≠ Application
For each security control found:
@@ -1099,10 +1097,16 @@ section, or gap shape alone.
exploitation, documentation deficiencies that don't enable exploit,
dead dependency pins, hardening recommendations without a concrete
exploit path, nice-to-have library improvements.
-- **Informational** — observations with security relevance that do not
- constitute vulnerabilities. Documentation gaps governed by
- foundation-level processes (e.g., ASF Security Team for ASF projects)
- rate Informational.
+- **Informational** — reserved for cases where a real, concrete code
+ defect exists with specific file and line references, but no clear
+ attack scenario can be constructed. This is the "downgrade-from-Low"
+ tier — a real bug without an exploit path. Do NOT use Informational
+ for requirements that don't apply to this component type, controls
+ delegated to other layers, or absences of features. Those cases are
+ N/A coverage, not findings. If your finding body would reasonably
+ contain "not applicable", "feature absent", "delegated to", or
+ "no [X] in this codebase", it belongs in N/A coverage and should
+ not be emitted as a finding.
### Severity calibration — apply per finding
@@ -1179,7 +1183,8 @@ For EACH ASVS requirement listed above, produce a section with this EXACT header
<controls relevant to this requirement, with location and coverage status>
### Positive Patterns for {section_id}
-<positive patterns specific to this requirement>
+For each positive pattern, format as: `- **<short name>**: <what's done correctly> — <file>:<line>`
+The trailing file:line reference is required so the consolidated report can cite the specific code under review.
```
Use the EXACT header `## ASVS-{section_id}:` (with the dash and colon) for each requirement section.
diff --git a/ASVS/agents/asvs_relevance_filter.py b/ASVS/agents/asvs_relevance_filter.py
index 7bf848c..9d602fa 100644
--- a/ASVS/agents/asvs_relevance_filter.py
+++ b/ASVS/agents/asvs_relevance_filter.py
@@ -1127,6 +1127,74 @@ async def run(input_dict, tools):
)
return any(p in desc for p in carve_phrases) and any(p in desc for p in did_phrases)
+ # Informational-absence filter: drop Informational findings that
+ # describe an absence ("not applicable", "no X in this codebase",
+ # "delegated to", etc.) rather than a real code defect. Per the
+ # severity calibration in the audit prompt, Informational is
+ # reserved for the downgrade-from-Low case — a real bug with no
+ # exploit path. Absences belong in N/A coverage, not the findings
+ # list. This catches Informational findings that slipped past the
+ # audit prompt's rule.
+ def _is_informational_absence(finding):
+ sev = (finding.get("severity") or "").strip().lower()
+ if sev not in ("info", "informational"):
+ return False
+
+ title = (finding.get("title") or "").lower()
+ desc = (finding.get("description") or "").lower()
+ body = f"{title}\n{desc}"
+ if not body.strip():
+ return False
+
+ # Phrase families that indicate absence rather than defect.
+ # Sourced empirically from absence-flavored Informational
+ # findings observed across audit runs.
+ absence_phrases = (
+ "not applicable",
+ "requirement not applicable",
+ "feature absent",
+ "feature is not present",
+ "not implemented",
+ "is not present in",
+ "no relevant code",
+ "out of scope",
+ "delegated to",
+ "absence of",
+ "no auth surface",
+ "no remote attack surface",
+ "registers no",
+ "opens no connections",
+ "no outbound calls",
+ "stub package",
+ "interface package",
+ "placeholder",
+ )
+ if any(p in body for p in absence_phrases):
+ return True
+
+ # "No X in/of/from Y" / "no X present" / "no X to inventory"
+ # pattern. Captures phrasings like "No multi-stage
+ # canonicalization", "No cookie operations in provided
+ # codebase", "No dynamic code execution primitives in the
+ # CLI". Word-boundary on "no" so it doesn't fire on "no
+ # clear attack scenario" or "no concrete exploit", which
+ # describe DEFECTS rather than absences.
+ import re
+ if re.search(
+ r"\bno\s+[a-z][a-z0-9_\- ]{2,40}\s+(in|of|from|present|to inventory|to validate|to review)\b",
+ body,
+ ):
+ return True
+
+ # An Informational finding without any concrete file
+ # reference is almost certainly an absence. Real
+ # defects-with-no-exploit point at code.
+ files = finding.get("files") or finding.get("affected_files") or []
+ if not files:
+ return True
+
+ return False
+
FILTER_PROMPT_TEMPLATE = (
"You are a security audit triage reviewer.\n\n"
"You have the project's own Security Profile (synthesized from the "
@@ -1361,6 +1429,41 @@ async def run(input_dict, tools):
f"{len(escape_hatch_drops)} defense-in-depth escape-hatch "
f"finding(s)")
+ # Informational-absence safety net. Audit prompt asks the
+ # model to route absences to N/A coverage rather than emit
+ # Info findings, but enforce structurally too in case any
+ # slip through. Absence-flavored Informational findings get
+ # moved from kept to dropped with confidence=high so they
+ # surface in the drop log but do not clutter the review
+ # queue (the absence regex/phrase set is empirical and
+ # well-tested, not speculative).
+ absence_drops = []
+ survivors = []
+ for f in kept:
+ if _is_informational_absence(f):
+ absence_drops.append(f)
+ else:
+ survivors.append(f)
+ if absence_drops:
+ kept = survivors
+ for f in absence_drops:
+ dropped.append({
+ "original_id": f.get("finding_id") or f.get("id") or "?",
+ "title": f.get("title", "?"),
+ "severity": f.get("severity", "?"),
+ "reason": (
+ "informational-absence: finding describes "
+ "non-applicability of a requirement rather than "
+ "a concrete code defect. Belongs in N/A coverage "
+ "per the audit prompt's severity calibration "
+ "rules. Filter dropped structurally."
+ ),
+ "confidence": "high",
+ })
+ print(f"[filter] {key}: post-pass dropped "
+ f"{len(absence_drops)} informational-absence "
+ f"finding(s)")
+
total_kept += len(kept)
total_dropped += len(dropped)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org