You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/06/30 00:14:18 UTC

(tooling-agents) branch main updated: asvs_consolidate: flag same-CWE/same-area findings with divergent severities

This is an automated email from the ASF dual-hosted git repository.

andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git


The following commit(s) were added to refs/heads/main by this push:
     new fe98330  asvs_consolidate: flag same-CWE/same-area findings with divergent severities
fe98330 is described below

commit fe983300b735a067ea761ab64bbc9cf49a83b83a
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Mon Jun 29 17:14:07 2026 -0700

    asvs_consolidate: flag same-CWE/same-area findings with divergent severities
    
    A Groovy audit rated an asSql() SQL-injection finding Critical while two
    sibling CWE-89 findings in the same subproject (groovy-sql) were rated Medium.
    The finding was real and correctly kept by the relevance filter; only the
    severity label was inflated by impact/CWE reflex. Nothing flagged the
    inconsistency for review.
    
    Add a deterministic calibration check after global_id assignment: group
    findings by (CWE, top-level code area = first 3 path segments of the primary
    affected file), and flag any group with 2+ findings spanning 2+ severities.
    Render the flagged groups as a Severity Calibration Review section in
    consolidated.md listing each group's findings and their severities.
    
    Review-only: does NOT change any severity (divergence is sometimes legitimate
    on exploitability grounds) — it surfaces the groups so an inflated or deflated
    label gets a human second look. Pure grouping + comparison, no model judgment.
    
    Tested against the Groovy finding set: flags the CWE-89 groovy-sql group
    (FINDING-001 Critical vs 003/004 Medium) and leaves the lone CWE-611 finding
    unflagged.
    
    This is the deterministic backstop for severity inflation. The judgment-side
    fixes (exploitability-aware severity rubric in the auditor; threat-model
    severity dampening) are tracked separately as enhancements.
    
    Single-agent update (redeploy asvs_consolidate only; no service restart).
---
 ASVS/agents/asvs_consolidate.py | 86 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)

diff --git a/ASVS/agents/asvs_consolidate.py b/ASVS/agents/asvs_consolidate.py
index ed58fe6..8c03948 100644
--- a/ASVS/agents/asvs_consolidate.py
+++ b/ASVS/agents/asvs_consolidate.py
@@ -1069,6 +1069,60 @@ If no duplicates: {"merges": []}"""
         for i, finding in enumerate(all_findings, 1):
             finding["global_id"] = f"FINDING-{i:03d}"
 
+        # ---- Severity calibration check -----------------------------------
+        # Catch the failure mode where the same vulnerability class in the
+        # same code area is rated inconsistently (e.g. one CWE-89 finding in
+        # groovy-sql at Critical sitting next to two CWE-89 findings in the
+        # same subproject at Medium). This does NOT auto-change severities --
+        # divergence is sometimes legitimate (different exploitability) -- it
+        # surfaces the groups for human review so an inflated/deflated label
+        # gets a second look. Deterministic: pure grouping + comparison, no
+        # model judgment. Groups by (CWE, top-level code area) where area is
+        # the first 3 path segments of the primary affected file, which keeps
+        # e.g. subprojects/groovy-sql together without lumping all of src.
+        severity_calibration_flags = []
+        try:
+            def _area_of(finding):
+                af = finding.get("affected_files") or []
+                if not af:
+                    return ""
+                first = af[0]
+                path = first.get("file", "") if isinstance(first, dict) else str(first)
+                return "/".join(path.split("/")[:3])
+
+            groups = {}
+            for f in all_findings:
+                cwe = (f.get("cwe") or "").strip()
+                area = _area_of(f)
+                if not cwe or not area:
+                    continue
+                groups.setdefault((cwe, area), []).append(f)
+
+            for (cwe, area), members in groups.items():
+                sevs = {m.get("severity", "Medium") for m in members}
+                if len(members) >= 2 and len(sevs) >= 2:
+                    ranked = sorted(
+                        members,
+                        key=lambda m: severity_order.get(m.get("severity", "Informational"), 4))
+                    severity_calibration_flags.append({
+                        "cwe": cwe,
+                        "area": area,
+                        "severities": sorted(sevs, key=lambda s: severity_order.get(s, 4)),
+                        "findings": [
+                            {"id": m["global_id"],
+                             "severity": m.get("severity", "Medium"),
+                             "title": m.get("title", "")}
+                            for m in ranked
+                        ],
+                    })
+            if severity_calibration_flags:
+                print(f"  Severity calibration: {len(severity_calibration_flags)} "
+                      f"CWE/area group(s) with divergent severities flagged for review",
+                      flush=True)
+        except Exception as _sc_e:
+            print(f"  WARN: severity calibration check failed "
+                  f"({type(_sc_e).__name__}: {_sc_e}); continuing", flush=True)
+
         # Persist the structured findings list so downstream agents (e.g.
         # asvs_compare_open_issues_prs) can read clean JSON instead of
         # re-parsing the rendered consolidated.md. This is the same data we
@@ -1506,6 +1560,38 @@ For multi-value table cells (Files, Source Reports, etc.): use ", " (comma-space
                 f"as GitHub issues; see issues.md for the {n_act} actionable items.*"
             )
 
+        # --- Severity Calibration Review (deterministic) ---
+        # Surfaces same-CWE / same-area findings that received different
+        # severities, so an inflated or deflated label gets a human second
+        # look. Does not change any severity; review-only.
+        if severity_calibration_flags:
+            tail_lines.append("")
+            tail_lines.append("---")
+            tail_lines.append("")
+            tail_lines.append("# Severity Calibration Review")
+            tail_lines.append("")
+            tail_lines.append(
+                "The following groups contain findings of the **same CWE in the "
+                "same code area** that were assigned **different severities**. "
+                "Divergence can be legitimate (genuinely different exploitability), "
+                "but it is also the classic shape of a miscalibrated label — e.g. "
+                "one finding rated Critical by impact reflex while its siblings of "
+                "the same class are Medium. Review each group and confirm the "
+                "spread is justified or align the outliers. *No severities were "
+                "changed automatically.*")
+            tail_lines.append("")
+            for fl in severity_calibration_flags:
+                tail_lines.append(
+                    f"### {fl['cwe']} in `{fl['area']}` — "
+                    f"severities: {', '.join(fl['severities'])}")
+                tail_lines.append("")
+                tail_lines.append("| Finding | Severity | Title |")
+                tail_lines.append("|---------|----------|-------|")
+                for m in fl["findings"]:
+                    title = (m["title"] or "").replace("|", "\\|")
+                    tail_lines.append(f"| {m['id']} | {m['severity']} | {title} |")
+                tail_lines.append("")
+
         tail_result = "\n".join(tail_lines)
 
         # Assemble


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org