You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/04/21 06:59:59 UTC

(tooling-agents) branch main updated: Remove ATR-isms, add download, discover, and consolidate to pipeline

This is an automated email from the ASF dual-hosted git repository.

andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git


The following commit(s) were added to refs/heads/main by this push:
     new d57e15e  Remove ATR-isms, add download, discover, and consolidate to pipeline
d57e15e is described below

commit d57e15e8edaae556e2961ed0ce83e4c2e6bee7d7
Author: Andrew Musselman <an...@gmail.com>
AuthorDate: Mon Apr 20 23:59:50 2026 -0700

    Remove ATR-isms, add download, discover, and consolidate to pipeline
---
 repos/apache/ASVS/README.md                        | 199 +++++
 .../add_markdown_file_to_github_directory.py       | 170 ++++
 .../consolidate_asvs_security_audit_reports.py     | 953 +++++++++++++++++++++
 .../ASVS/agents/discover_codebase_architecture.py  | 366 ++++++++
 .../agents/download_github_repo_to_data_store.py   | 149 ++++
 .../agents/orchestrate_asvs_audit_to_github.py     | 368 ++++++++
 .../apache/ASVS/agents/run_asvs_security_audit.py  | 947 ++++++++++++++++++++
 7 files changed, 3152 insertions(+)

diff --git a/repos/apache/ASVS/README.md b/repos/apache/ASVS/README.md
new file mode 100644
index 0000000..e463c22
--- /dev/null
+++ b/repos/apache/ASVS/README.md
@@ -0,0 +1,199 @@
+# ASVS Security Audit Pipeline
+
+## Pipeline Flow
+
+```
+orchestrate_asvs_audit_to_github         (single entry point)
+  │
+  ├──▶ download_github_repo_to_datastore  (once — fetches source code)
+  │
+  ├──▶ discover_codebase_architecture     (once — generates audit plan)
+  │
+  ├──▶ run_asvs_security_audit            (× N sections)
+  │    └──▶ add_markdown_file_to_github_directory  (× N)
+  │
+  └──▶ consolidate_asvs_security_audit_reports  (once — final report)
+```
+
+Pre-requisites (one-time, outside the pipeline):
+- ASVS requirements loaded into the `asvs` data store namespace
+- Optionally: audit guidance loaded via `fetch_audit_guidance`
+
+## Quick Start
+
+Run `orchestrate_asvs_audit_to_github` with:
+
+| Input | Value |
+|---|---|
+| `sourceRepo` | `apache/airflow` |
+| `outputRepo` | `apache/tooling-runbooks` |
+| `outputToken` | `ghp_...` |
+| `outputDirectory` | `ASVS/reports` |
+| `level` | `L2` |
+| `severityThreshold` | `HIGH` |
+
+That's it. The orchestrator downloads the code, fetches the latest commit
+hash, discovers the architecture, runs all audits, pushes reports, and
+consolidates. The output directory is automatically extended with the
+repo name and commit hash: `ASVS/reports/airflow/da901ba`.
+
+To audit a subdirectory of a large repo:
+
+| Input | Value |
+|---|---|
+| `sourceRepo` | `apache/airflow/airflow-core/src` |
+
+This downloads only files under `airflow-core/src/`.
+
+## Reports Location
+
+```
+ASVS/reports/airflow/da901ba/       ← auto-generated from repo + HEAD commit
+├── auth_identity/                  ← per-domain subdirectories
+│   ├── 6.1.1.md
+│   └── 7.1.1.md
+├── secrets_crypto/
+│   └── 9.1.1.md
+├── consolidated.md                 ← THE REPORT
+└── issues.md                       ← GitHub issues, one per finding
+```
+
+## Level System
+
+| Input | Requirements audited |
+|---|---|
+| `L1` | L1 only |
+| `L2` | L1 + L2 |
+| `L3` | L1 + L2 + L3 (all) |
+
+## Severity Threshold
+
+| Input | Findings included |
+|---|---|
+| `CRITICAL` | Critical only |
+| `HIGH` | Critical + High |
+| `MEDIUM` | Critical + High + Medium |
+| (empty) | All findings |
+
+---
+
+## Agent Reference
+
+All gofannon inputs are strings.
+
+### 1. orchestrate_asvs_audit_to_github
+
+The main entry point. Calls all other agents.
+
+| Input | Required | Description |
+|---|---|---|
+| `sourceRepo` | yes | Source code to audit. Accepts `owner/repo`, `owner/repo/subdir`, or full GitHub URL like `https://github.com/owner/repo/tree/branch/subdir` |
+| `sourceToken` | no | PAT for private source repos |
+| `supplementalData` | no | Extra data store namespaces, comma-separated (e.g., `audit_guidance`) |
+| `outputRepo` | yes | GitHub repo for reports (`owner/repo`) |
+| `outputToken` | yes | PAT with write access to output repo |
+| `outputDirectory` | yes | Base directory — repo name and commit hash are appended automatically |
+| `discover` | no | `"true"` or `"false"` (default `"true"`) |
+| `level` | no | `"L1"`, `"L2"`, or `"L3"` |
+| `severityThreshold` | no | `"CRITICAL"`, `"HIGH"`, `"MEDIUM"`, or empty |
+| `consolidate` | no | `"true"` or `"false"` (default `"true"`) |
+
+**Output:** `outputText` — combined audit results.
+
+**Namespace derivation:** The orchestrator parses `sourceRepo` to derive the code namespace automatically. `apache/airflow` → `files:apache/airflow`. If `supplementalData` is `audit_guidance`, the namespace list becomes `["files:apache/airflow", "audit_guidance"]`.
+
+**Agents called:**
+1. `download_github_repo_to_datastore` — downloads source code
+2. `discover_codebase_architecture` — generates audit plan (if `discover="true"`)
+3. `run_asvs_security_audit` — once per ASVS section
+4. `add_markdown_file_to_github_directory` — once per section
+5. `consolidate_asvs_security_audit_reports` — final report (if `consolidate="true"`)
+
+---
+
+### 2. download_github_repo_to_datastore
+
+Downloads a GitHub repo (or subdirectory) into the data store.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | `owner/repo` or `owner/repo/subdir`, optional token on next line |
+
+**Output:** `outputText` — download summary.
+
+Stores files in namespace `files:{owner}/{repo}`. When a path prefix is included (e.g., `apache/airflow/airflow-core/src`), only files under that path are downloaded. File paths in the data store are preserved as full repo-relative paths.
+
+---
+
+### 3. discover_codebase_architecture
+
+Scans codebase, generates domains + file lists + false positive guidance.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputNamespace` | yes | Data store namespace(s) with code, comma-separated |
+
+**Output:** `outputText` — JSON passConfig containing `passes`, `domain_groups`, `false_positive_guidance`.
+
+---
+
+### 4. run_asvs_security_audit
+
+Audits code against a single ASVS requirement.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | JSON string (see fields below) |
+
+JSON fields inside `inputText`:
+
+| Field | Required | Description |
+|---|---|---|
+| `namespaces` | yes | Array of data store namespaces |
+| `asvs` | yes | ASVS section (e.g., `"6.1.1"`) |
+| `includeFiles` | no | Array of file glob patterns — skips relevance filtering |
+| `domainContext` | no | Architecture context for Opus prompt |
+| `severityThreshold` | no | Minimum severity to report |
+| `falsePositiveGuidance` | no | Array of patterns to suppress |
+
+**Output:** `outputText` — markdown audit report.
+
+---
+
+### 5. consolidate_asvs_security_audit_reports
+
+Reads per-section reports from GitHub, deduplicates, produces consolidated report + issues.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | Multi-line: `repo:`, `pat:`, `directories:`, `output:` |
+| `domainGroups` | no | JSON string mapping domain names to ASVS section arrays |
+| `level` | no | `"L1"`, `"L2"`, or `"L3"` (default `"L3"`) |
+
+**Output:** `outputText` — summary. Pushes `consolidated.md` and `issues.md` to GitHub.
+
+---
+
+### 6. add_markdown_file_to_github_directory
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | JSON with `repo`, `token`, `directory`, `filename` |
+| `commitMessage` | yes | Git commit message |
+| `fileContents` | yes | Markdown content |
+
+**Output:** `outputText` — GitHub API response.
+
+---
+
+## Troubleshooting
+
+**"sourceRepo is required"** — Provide `sourceRepo` as `owner/repo`, `owner/repo/subdir`, or a full GitHub URL.
+
+**"No ASVS sections match level L1"** — Check that ASVS requirement entries have a `level` field.
+
+**Download takes too long** — Use a path prefix to scope: `apache/airflow/airflow-core/src` instead of `apache/airflow`.
+
+**Too many findings** — Use `severityThreshold="HIGH"` or `"CRITICAL"`.
+
+**Reports not on GitHub** — Check `outputToken` has write access to `outputRepo`.
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/add_markdown_file_to_github_directory.py b/repos/apache/ASVS/agents/add_markdown_file_to_github_directory.py
new file mode 100644
index 0000000..4d29665
--- /dev/null
+++ b/repos/apache/ASVS/agents/add_markdown_file_to_github_directory.py
@@ -0,0 +1,170 @@
+# add_markdown_file_to_github_directory
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+import litellm
+import httpx
+
+async def run(input_dict, tools):
+    mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+    http_client = httpx.AsyncClient()
+    try:
+        import json
+        import re
+        import base64
+        from datetime import datetime, timezone
+
+        input_text = (input_dict.get("inputText") or "").strip()
+        commit_message = input_dict.get("commitMessage") or "Add markdown file"
+        file_contents = input_dict.get("fileContents") or ""
+
+        def _maybe_json(s: str):
+            try:
+                return json.loads(s)
+            except Exception:
+                return None
+
+        def _strip_quotes(s: str) -> str:
+            s = s.strip()
+            if (len(s) >= 2) and ((s[0] == s[-1]) and s[0] in ("'", '"')):
+                return s[1:-1].strip()
+            return s
+
+        def _find_kv(text: str, key: str):
+            patterns = [
+                rf'(?im)^\s*{re.escape(key)}\s*:\s*(.+?)\s*$',
+                rf'(?im)^\s*{re.escape(key)}\s*=\s*(.+?)\s*$',
+                rf'(?is)"{re.escape(key)}"\s*:\s*"([^"]+)"',
+                rf"(?is)'{re.escape(key)}'\s*:\s*'([^']+)'",
+            ]
+            for pat in patterns:
+                m = re.search(pat, text)
+                if m:
+                    return _strip_quotes(m.group(1))
+            return None
+
+        def _parse_owner_repo(text: str):
+            repo = _find_kv(text, "repo") or _find_kv(text, "repository") or _find_kv(text, "full_name")
+            if repo and "/" in repo:
+                parts = repo.strip().strip("/").split("/")
+                if len(parts) >= 2:
+                    return parts[-2], parts[-1]
+            m = re.search(r'(?i)\bgithub\.com/([^/\s]+)/([^/\s#?]+)', text)
+            if m:
+                return m.group(1), m.group(2)
+            m = re.search(r'(?i)\b([A-Za-z0-9_.-]+)/([A-Za-z0-9_.-]+)\b', text)
+            if m:
+                return m.group(1), m.group(2)
+            return None, None
+
+        cfg = _maybe_json(input_text)
+        if not isinstance(cfg, dict):
+            cfg = {}
+
+        token = (
+            cfg.get("token") or cfg.get("github_token") or cfg.get("githubToken")
+            or _find_kv(input_text, "token") or _find_kv(input_text, "github_token")
+            or _find_kv(input_text, "githubToken")
+        )
+
+        api_base = (
+            cfg.get("api_base") or cfg.get("apiBase") or cfg.get("github_api_base")
+            or _find_kv(input_text, "api_base") or _find_kv(input_text, "apiBase")
+            or _find_kv(input_text, "github_api_base") or "https://api.github.com"
+        ).rstrip("/")
+
+        owner = cfg.get("owner") or _find_kv(input_text, "owner")
+        repo_name = cfg.get("repo_name") or cfg.get("repo") or cfg.get("repository") or _find_kv(input_text, "repo_name")
+        if repo_name and isinstance(repo_name, str) and "/" in repo_name and not owner:
+            owner, repo_name = repo_name.split("/", 1)
+
+        if not owner or not repo_name:
+            o2, r2 = _parse_owner_repo(input_text)
+            owner = owner or o2
+            repo_name = repo_name or r2
+
+        directory = (
+            cfg.get("directory") or cfg.get("dir") or cfg.get("path")
+            or _find_kv(input_text, "directory") or _find_kv(input_text, "dir")
+            or _find_kv(input_text, "path") or ""
+        ).strip()
+
+        branch = cfg.get("branch") or _find_kv(input_text, "branch") or None
+
+        file_name = (
+            cfg.get("fileName") or cfg.get("filename") or cfg.get("file_name")
+            or _find_kv(input_text, "fileName") or _find_kv(input_text, "filename")
+            or _find_kv(input_text, "file_name")
+        )
+
+        file_path = (
+            cfg.get("filePath") or cfg.get("filepath")
+            or _find_kv(input_text, "filePath") or _find_kv(input_text, "filepath")
+        )
+
+        if not token:
+            return {"outputText": "Error: No GitHub token found in inputText (expected keys like token/github_token)."}
+        if not owner or not repo_name:
+            return {"outputText": "Error: Could not determine GitHub owner/repo from inputText (expected owner+repo or owner/repo)."}
+        if not directory and not file_path:
+            return {"outputText": "Error: No target directory/path provided in inputText (expected directory/dir/path or filePath)."}
+        if not file_contents:
+            return {"outputText": "Error: fileContents is empty."}
+
+        if file_path:
+            final_path = file_path.strip().lstrip("/")
+        else:
+            dir_clean = directory.strip().strip("/")
+            if file_name:
+                name = file_name.strip().lstrip("/")
+            else:
+                ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+                name = f"new_file_{ts}.md"
+            if not name.lower().endswith(".md"):
+                name = f"{name}.md"
+            final_path = f"{dir_clean}/{name}" if dir_clean else name
+
+        headers = {
+            "Authorization": f"token {token}",
+            "Accept": "application/vnd.github+json",
+        }
+
+        get_url = f"{api_base}/repos/{owner}/{repo_name}/contents/{final_path}"
+        params = {}
+        if branch:
+            params["ref"] = branch
+
+        existing_sha = None
+        existing_resp = await http_client.get(get_url, headers=headers, params=params)
+        if existing_resp.status_code == 200:
+            try:
+                existing_json = existing_resp.json()
+                existing_sha = existing_json.get("sha")
+            except Exception:
+                pass
+
+        put_url = f"{api_base}/repos/{owner}/{repo_name}/contents/{final_path}"
+        payload = {
+            "message": commit_message,
+            "content": base64.b64encode(file_contents.encode("utf-8")).decode("utf-8"),
+        }
+        if existing_sha:
+            payload["sha"] = existing_sha
+        if branch:
+            payload["branch"] = branch
+
+        resp = await http_client.put(put_url, headers=headers, json=payload)
+        try:
+            resp.raise_for_status()
+        except Exception:
+            try:
+                return {"outputText": json.dumps(resp.json(), indent=2, sort_keys=True)}
+            except Exception:
+                return {"outputText": resp.text}
+
+        try:
+            return {"outputText": json.dumps(resp.json(), indent=2, sort_keys=True)}
+        except Exception:
+            return {"outputText": resp.text}
+
+    finally:
+        await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/consolidate_asvs_security_audit_reports.py b/repos/apache/ASVS/agents/consolidate_asvs_security_audit_reports.py
new file mode 100644
index 0000000..c54c76d
--- /dev/null
+++ b/repos/apache/ASVS/agents/consolidate_asvs_security_audit_reports.py
@@ -0,0 +1,953 @@
+# consolidate_asvs_security_audit_reports
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+async def run(input_dict, tools):
+    mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+    http_client = httpx.AsyncClient()
+    try:
+        import json
+        import re
+        import base64
+        from datetime import date
+
+        audit_date = date.today().strftime("%b %d, %Y")
+
+        # =============================================================
+        # Markdown HTML sanitizer
+        # =============================================================
+        def sanitize_md_html(text):
+            if not text:
+                return text
+            parts = re.split(r'(```[\s\S]*?```)', text)
+            out = []
+            for part in parts:
+                if part.startswith('```'):
+                    out.append(part)
+                    continue
+                stash = []
+                def _stash(m):
+                    stash.append(m.group(0))
+                    return f'\x00IC{len(stash)-1}\x00'
+                s = re.sub(r'`[^`\n]+`', _stash, part)
+                s = re.sub(r'<(/?\w[^>]*)>', r'&lt;\1&gt;', s)
+                for j, code in enumerate(stash):
+                    s = s.replace(f'\x00IC{j}\x00', code)
+                out.append(s)
+            return ''.join(out)
+
+        # =============================================================
+        # Parse inputs
+        # =============================================================
+        input_text = input_dict.get("inputText", "")
+        domain_groups_raw = input_dict.get("domainGroups", "")
+        level = input_dict.get("level", "L3")
+
+        # Normalize level
+        level = level.strip().upper()
+        if level and not level.startswith("L"):
+            level = f"L{level}"
+        LEVEL_ORDER = {"L1": 1, "L2": 2, "L3": 3}
+        max_level_num = LEVEL_ORDER.get(level, 3)
+
+        print(f"DEBUG raw input: {repr(input_text[:500])}", flush=True)
+        lines = input_text.strip().split("\n")
+        owner_repo = ""
+        pat = ""
+        directories_raw = ""
+        output_directory = ""
+        for line in lines:
+            line = line.strip()
+            if not line:
+                continue
+            if ":" in line:
+                key, _, value = line.partition(":")
+                key = key.strip().lower()
+                value = value.strip()
+                if key in ("owner/repo", "repo", "repository", "owner_repo"):
+                    owner_repo = value
+                elif key in ("pat", "token", "github_token", "personal_access_token"):
+                    pat = value
+                elif key in ("directories", "dirs", "paths", "directory", "dir", "path"):
+                    directories_raw = value
+                elif key in ("output", "output_directory", "output_dir"):
+                    output_directory = value
+
+        directories = [d.strip().strip("/") for d in directories_raw.split(",") if d.strip()]
+
+        if not owner_repo or not pat or not directories:
+            parts = input_text.strip().split()
+            tokens = [p.strip() for p in parts if p.strip()]
+            if not owner_repo:
+                for t in tokens:
+                    if "/" in t and not t.startswith("/") and not t.startswith("ghp_") and "." not in t.split("/")[0]:
+                        owner_repo = t
+                        break
+            if not pat:
+                for t in tokens:
+                    if t.startswith("ghp_") or t.startswith("github_pat_"):
+                        pat = t
+                        break
+
+        assert owner_repo, "Could not parse owner/repo from input"
+        assert pat, "Could not parse PAT from input"
+        assert directories, "Could not parse directories from input"
+
+        owner, repo = owner_repo.split("/", 1)
+
+        if not output_directory:
+            output_directory = "/".join(directories[0].split("/")[:-1])
+        output_directory = output_directory.strip("/")
+
+        # Output filenames — no level suffix
+        consolidated_filename = "consolidated.md"
+        issues_filename = "issues.md"
+
+        print(f"Repository: {owner}/{repo}")
+        print(f"Directories: {directories}")
+        print(f"Level: {level}")
+        print(f"Output: {output_directory}")
+
+        # =============================================================
+        # Model configuration
+        # =============================================================
+        FAST_PROVIDER = "bedrock"
+        FAST_MODEL = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+        FAST_PARAMS = {"temperature": 0.7, "max_tokens": 16384}
+
+        HEAVY_PROVIDER = "bedrock"
+        HEAVY_MODEL = "us.anthropic.claude-opus-4-6-v1"
+        HEAVY_PARAMS = {"temperature": 1, "reasoning_effort": "high", "max_tokens": 128000}
+
+        FAST_CONTEXT_WINDOW = get_context_window(FAST_PROVIDER, FAST_MODEL)
+        HEAVY_CONTEXT_WINDOW = get_context_window(HEAVY_PROVIDER, HEAVY_MODEL)
+
+        dirs_key = "+".join(sorted(directories))
+        extraction_ns = data_store.use_namespace(f"extraction:{owner}/{repo}/{dirs_key}")
+        consolidation_ns = data_store.use_namespace(f"consolidation:{owner}/{repo}/{dirs_key}")
+
+        GITHUB_API = "https://api.github.com"
+        headers = {
+            "Authorization": f"token {pat}",
+            "Accept": "application/vnd.github.v3+json",
+            "X-GitHub-Api-Version": "2022-11-28",
+        }
+        repo_resp = await http_client.get(f"{GITHUB_API}/repos/{owner}/{repo}", headers=headers)
+        repo_data = repo_resp.json()
+        default_branch = repo_data.get("default_branch", "main")
+
+        # ============================================================
+        # PHASE 1: Read All Reports from All Directories
+        # ============================================================
+        print("\n=== PHASE 1: Reading all reports ===")
+
+        reports = {}
+
+        for directory in directories:
+            print(f"\n  Reading reports from {directory}...")
+            contents_resp = await http_client.get(
+                f"{GITHUB_API}/repos/{owner}/{repo}/contents/{directory}",
+                headers=headers,
+                params={"ref": default_branch},
+            )
+            if contents_resp.status_code != 200:
+                print(f"  WARNING: Failed to list {directory}: {contents_resp.status_code}")
+                continue
+
+            dir_contents = contents_resp.json()
+            report_files = []
+            for item in dir_contents:
+                if item["type"] == "file" and item["name"].endswith(".md"):
+                    if not item["name"].startswith("consolidated") and \
+                       not item["name"].startswith("issues"):
+                        report_files.append(item)
+
+            print(f"  Found {len(report_files)} report files")
+
+            for item in report_files:
+                file_resp = await http_client.get(
+                    f"{GITHUB_API}/repos/{owner}/{repo}/contents/{directory}/{item['name']}",
+                    headers=headers,
+                    params={"ref": default_branch},
+                )
+                if file_resp.status_code == 200:
+                    file_data = file_resp.json()
+                    content = base64.b64decode(file_data["content"]).decode("utf-8", errors="replace")
+                    report_key = item['name']
+                    reports[report_key] = content
+                    print(f"    Read {report_key} ({len(content)} chars)")
+
+        total_reports = len(reports)
+        print(f"\nSuccessfully read {total_reports} reports")
+
+        # ============================================================
+        # PHASE 2: Extract Findings (Sonnet, parallel, checkpointed)
+        # ============================================================
+        print("\n=== PHASE 2: Extracting findings ===")
+
+        EXTRACTION_PROMPT_TEMPLATE = """You are a security finding extractor. Given an ASVS audit report, extract ALL findings into structured JSON.
+
+For each finding, capture:
+- source_report: the report filename
+- finding_id: the ID used in the source report
+- severity: Critical/High/Medium/Low/Informational
+- title: short descriptive title
+- description: full description of the vulnerability
+- cwe: CWE identifier if mentioned
+- asvs_section: the ASVS section being audited (e.g., "8.2.2")
+- affected_files: list of objects with "file" and "line" keys
+- recommended_remediation: the recommended fix
+- positive_controls: list of any positive security controls noted
+
+Also extract:
+- asvs_status: overall status of this ASVS section (Pass/Fail/Partial/N/A)
+- asvs_section_title: the title/topic of this ASVS section if mentioned
+
+If the report has NO findings, return an empty findings list but still set asvs_status.
+
+Return ONLY valid JSON:
+{
+  "source_report": "filename.md",
+  "asvs_section": "X.Y.Z",
+  "asvs_section_title": "Title",
+  "asvs_status": "Pass|Fail|Partial|N/A",
+  "findings": [...],
+  "positive_controls": [{"control": "description", "evidence": "where observed", "files": ["file:line"]}]
+}"""
+
+        extraction_semaphore = asyncio.Semaphore(5)
+        all_extracted = {}
+        extraction_errors = []
+
+        async def extract_report(report_key, content):
+            cached = extraction_ns.get(report_key)
+            if cached:
+                print(f"  {report_key}: cached ({len(cached.get('findings', []))} findings)")
+                return report_key, cached, None
+
+            async with extraction_semaphore:
+                print(f"  {report_key}: extracting...", end=" ", flush=True)
+                messages = [
+                    {"role": "user", "content": f"{EXTRACTION_PROMPT_TEMPLATE}\n\nSource report: {report_key}\n\nReport content:\n{content}"}
+                ]
+                try:
+                    result, _ = await call_llm(
+                        provider=FAST_PROVIDER, model=FAST_MODEL,
+                        messages=messages, parameters=FAST_PARAMS, timeout=600,
+                    )
+                    json_match = re.search(r'\{[\s\S]*\}', result)
+                    if json_match:
+                        extracted = json.loads(json_match.group())
+                        extracted["source_report"] = report_key
+                        extraction_ns.set(report_key, extracted)
+                        print(f"{len(extracted.get('findings', []))} findings, status: {extracted.get('asvs_status', '?')}")
+                        return report_key, extracted, None
+                    else:
+                        print(f"WARNING: no JSON")
+                        return report_key, None, "No JSON in result"
+                except Exception as e:
+                    print(f"ERROR: {e}")
+                    return report_key, None, str(e)
+
+        extraction_results = await asyncio.gather(*[
+            extract_report(rk, content) for rk, content in reports.items()
+        ])
+        for rk, extracted, error in extraction_results:
+            if extracted:
+                all_extracted[rk] = extracted
+            elif error:
+                extraction_errors.append(rk)
+
+        if extraction_errors:
+            print(f"\nRetrying {len(extraction_errors)} failed extractions...")
+            retry_results = await asyncio.gather(*[
+                extract_report(rk, reports[rk]) for rk in extraction_errors
+            ])
+            for rk, extracted, error in retry_results:
+                if extracted:
+                    all_extracted[rk] = extracted
+                    extraction_errors.remove(rk)
+
+        total_extracted = sum(len(v.get("findings", [])) for v in all_extracted.values())
+        print(f"\nTotal extracted findings: {total_extracted}")
+
+        # ============================================================
+        # PHASE 2.5: Enrich with ASVS requirement context
+        # ============================================================
+        print("\n=== Loading ASVS requirement context ===")
+
+        asvs_ns = data_store.use_namespace("asvs")
+        ASVS_PREFIX = "asvs:"
+
+        all_asvs_ids = set()
+        for extracted in all_extracted.values():
+            section = extracted.get("asvs_section", "")
+            if section:
+                all_asvs_ids.add(section)
+            for finding in extracted.get("findings", []):
+                s = finding.get("asvs_section", "")
+                if s:
+                    all_asvs_ids.add(s)
+
+        asvs_context = {}
+        chapters_cache = {}
+        sections_cache = {}
+
+        for req_id in sorted(all_asvs_ids):
+            req = asvs_ns.get(f"{ASVS_PREFIX}requirements:{req_id}")
+            if not req:
+                continue
+            section_id = req.get("section_id", "")
+            chapter_id = req.get("chapter_id", "")
+            if section_id and section_id not in sections_cache:
+                sections_cache[section_id] = asvs_ns.get(f"{ASVS_PREFIX}sections:{section_id}")
+            if chapter_id and chapter_id not in chapters_cache:
+                chapters_cache[chapter_id] = asvs_ns.get(f"{ASVS_PREFIX}chapters:{chapter_id}")
+            sec = sections_cache.get(section_id) or {}
+            ch = chapters_cache.get(chapter_id) or {}
+            asvs_context[req_id] = {
+                "req_description": req.get("req_description", ""),
+                "level": req.get("level"),
+                "section_name": sec.get("section_name", ""),
+                "section_description": sec.get("description", ""),
+                "chapter_name": ch.get("chapter_name", ""),
+                "control_objective": ch.get("control_objective", ""),
+            }
+
+        print(f"  Loaded ASVS context for {len(asvs_context)} requirements")
+
+        def get_asvs_level(section_id):
+            ctx = asvs_context.get(section_id)
+            if ctx and ctx.get("level"):
+                return f"L{ctx['level']}"
+            return "L?"
+
+        # ============================================================
+        # PHASE 3: Domain-Grouped Consolidation
+        # ============================================================
+        print("\n=== PHASE 3: Domain-grouped consolidation ===")
+
+        # Domain groups from input or ATR defaults
+        if domain_groups_raw:
+            DOMAIN_GROUPS = json.loads(domain_groups_raw) if isinstance(domain_groups_raw, str) else domain_groups_raw
+            print(f"  Using provided domain groups: {len(DOMAIN_GROUPS)} domains", flush=True)
+        else:
+            DOMAIN_GROUPS = {
+                "input_encoding": ["1.1.1","1.1.2","1.2.1","1.2.2","1.2.3","1.2.4","1.2.5","1.3.1","1.3.2","1.4.1","1.4.2","1.4.3","1.5.1"],
+                "business_logic": ["2.1.1","2.2.1","2.2.2","2.3.1","2.4.1"],
+                "session_csrf": ["3.2.1","3.2.2","3.3.1","3.4.1","3.4.2","3.5.1","3.5.2","3.5.3","3.7.1","3.7.2"],
+                "content_type": ["4.1.1","4.2.1","4.3.1","4.3.2","4.4.1"],
+                "file_path": ["5.1.1","5.2.1","5.2.2","5.3.1","5.3.2","5.4.1","5.4.2","5.4.3"],
+                "auth_rate_limit": ["6.1.1","6.2.1","6.2.2","6.2.3","6.2.4","6.2.5","6.2.6","6.2.7","6.2.8","6.3.1","6.3.2","6.4.1","6.4.2","6.5.1","6.5.2","6.5.3","6.5.4","6.5.5","6.6.1","6.6.2","6.6.3","6.8.1","6.8.2","6.8.3","6.8.4"],
+                "session_token": ["7.1.1","7.1.2","7.1.3","7.2.1","7.2.2","7.2.3","7.2.4","7.3.1","7.3.2","7.4.1","7.4.2","7.5.1","7.5.2","7.6.1","7.6.2"],
+                "authorization": ["8.1.1","8.2.1","8.2.2","8.3.1","8.4.1"],
+                "jwt_token": ["9.1.1","9.1.2","9.1.3","9.2.1"],
+                "oauth": ["10.1.1","10.1.2","10.2.1","10.2.2","10.3.1","10.3.2","10.3.3","10.3.4","10.4.1","10.4.2","10.4.3","10.4.4","10.4.5","10.5.1","10.5.2","10.5.3","10.5.4","10.5.5","10.6.1","10.6.2","10.7.1","10.7.2","10.7.3"],
+                "crypto_tls": ["11.1.1","11.1.2","11.2.1","11.2.2","11.2.3","11.3.1","11.3.2","11.4.1","11.5.1","11.6.1","12.1.1","12.2.1","12.2.2","12.3.1","12.3.2","12.3.3","12.3.4"],
+                "api_scm_client": ["13.1.1","13.2.1","13.2.2","13.2.3","13.2.4","13.2.5","13.3.1","13.3.2","13.4.1","14.1.1","14.1.2","14.2.1","14.3.1"],
+                "dependencies": ["15.1.1","15.2.1","15.3.1"],
+                "audit_logging": ["16.1.1","16.2.1","16.2.2","16.2.3","16.2.4","16.2.5","16.3.1","16.3.2","16.3.3","16.3.4","16.4.1","16.4.2","16.4.3","16.5.1","16.5.2","16.5.3"],
+                "webrtc": ["17.1.1","17.2.1","17.2.2","17.2.3","17.2.4","17.3.1","17.3.2"],
+            }
+            print(f"  Using default ATR domain groups: {len(DOMAIN_GROUPS)} domains", flush=True)
+
+        section_to_domain = {}
+        for domain, sections in DOMAIN_GROUPS.items():
+            for section in sections:
+                section_to_domain[section] = domain
+
+        chapter_domain_map = {}
+        for domain, sections in DOMAIN_GROUPS.items():
+            for section in sections:
+                chapter = ".".join(section.split(".")[:2])
+                chapter_domain_map[chapter] = domain
+
+        domain_reports = {domain: {} for domain in DOMAIN_GROUPS}
+        domain_reports["misc"] = {}
+
+        for report_key, extracted in all_extracted.items():
+            asvs_section = extracted.get("asvs_section", "")
+            if not asvs_section:
+                filename_part = report_key.split(":")[-1] if ":" in report_key else report_key
+                match = re.match(r'(\d+\.\d+\.\d+)', filename_part)
+                if match:
+                    asvs_section = match.group(1)
+
+            domain = section_to_domain.get(asvs_section)
+            if not domain and asvs_section:
+                chapter = ".".join(asvs_section.split(".")[:2])
+                domain = chapter_domain_map.get(chapter, "misc")
+            if not domain:
+                domain = "misc"
+
+            if domain not in domain_reports:
+                domain_reports[domain] = {}
+            domain_reports[domain][report_key] = extracted
+
+        domain_reports = {d: r for d, r in domain_reports.items() if r}
+
+        for domain, rpts in domain_reports.items():
+            finding_count = sum(len(r.get("findings", [])) for r in rpts.values())
+            print(f"  {domain}: {len(rpts)} reports, {finding_count} findings")
+
+        CONSOLIDATION_PROMPT = f"""You are a security audit consolidator. You are given extracted findings from multiple ASVS audit reports within the SAME security domain.
+
+Audit scope: up to {level}
+
+Your job is to:
+1. **Identify TRUE duplicates**: The EXACT same vulnerability in the EXACT same code location. Merge these into ONE finding.
+2. **Preserve EVERY unique finding**. If in doubt, keep findings SEPARATE.
+3. **Track ASVS levels**: Each finding must list which ASVS level(s) apply based on its ASVS sections.
+4. **Use the ASVS requirement descriptions** provided to understand what each section tests for.
+5. **Do NOT add cross-references between findings.**
+
+**Deduplication test**: If a developer could fix one WITHOUT fixing the other, they are SEPARATE findings.
+
+Return valid JSON:
+{{
+  "domain": "domain_name",
+  "consolidated_findings": [
+    {{
+      "temp_id": "DOMAIN-N",
+      "severity": "Critical|High|Medium|Low|Informational",
+      "title": "descriptive title",
+      "description": "full description",
+      "cwe": "CWE-NNN or null",
+      "asvs_sections": ["X.Y.Z", ...],
+      "asvs_levels": ["L1", "L2"],
+      "affected_files": [{{"file": "path", "line": "N"}}],
+      "source_reports": ["filename.md", ...],
+      "recommended_remediation": "specific fix",
+      "merged_from": ["original finding IDs"]
+    }}
+  ],
+  "positive_controls": [
+    {{"control": "description", "evidence": "where observed", "files": ["file:line"]}}
+  ],
+  "asvs_statuses": {{
+    "X.Y.Z": {{"status": "Pass|Fail|Partial|N/A", "title": "section title"}}
+  }},
+  "dedup_log": [
+    "Merged FINDING-X with FINDING-Y because: reason"
+  ]
+}}"""
+
+        consolidation_semaphore = asyncio.Semaphore(3)
+        domain_consolidated = {}
+
+        def build_asvs_context_block(rpts):
+            domain_asvs_sections = set()
+            for rpt_data in rpts.values():
+                s = rpt_data.get("asvs_section", "")
+                if s:
+                    domain_asvs_sections.add(s)
+            if not domain_asvs_sections:
+                return ""
+            ctx_lines = []
+            for sid in sorted(domain_asvs_sections):
+                ctx = asvs_context.get(sid)
+                if ctx:
+                    ctx_lines.append(
+                        f"- **{sid}** ({get_asvs_level(sid)}, {ctx['section_name']}): "
+                        f"{ctx['req_description'][:300]}"
+                    )
+            if not ctx_lines:
+                return ""
+            return "\n\n## ASVS Requirement Descriptions\n" + "\n".join(ctx_lines)
+
+        async def consolidate_domain(domain, rpts):
+            cached = consolidation_ns.get(domain)
+            if cached:
+                print(f"  {domain}: cached ({len(cached.get('consolidated_findings', []))} findings)")
+                return domain, cached
+
+            async with consolidation_semaphore:
+                print(f"  {domain}: consolidating {len(rpts)} reports...", flush=True)
+                domain_data = json.dumps(rpts, indent=2, default=str)
+                asvs_block = build_asvs_context_block(rpts)
+                user_msg = f"Domain: {domain}{asvs_block}\n\nExtracted findings:\n\n{domain_data}"
+                messages = [{"role": "user", "content": f"{CONSOLIDATION_PROMPT}\n\n{user_msg}"}]
+
+                msg_tokens = count_message_tokens(messages, FAST_PROVIDER, FAST_MODEL)
+                limit = int(FAST_CONTEXT_WINDOW * 0.80)
+
+                if msg_tokens > limit:
+                    items = list(rpts.items())
+                    mid = len(items) // 2
+                    sub_results = []
+                    for si, sub in enumerate([dict(items[:mid]), dict(items[mid:])]):
+                        sub_data = json.dumps(sub, indent=2, default=str)
+                        sub_asvs = build_asvs_context_block(sub)
+                        sub_msg = f"Domain: {domain} (sub-group {si+1}/2){sub_asvs}\n\nFindings:\n\n{sub_data}"
+                        sub_messages = [{"role": "user", "content": f"{CONSOLIDATION_PROMPT}\n\n{sub_msg}"}]
+                        try:
+                            result, _ = await call_llm(
+                                provider=FAST_PROVIDER, model=FAST_MODEL,
+                                messages=sub_messages,
+                                parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=600,
+                            )
+                            json_match = re.search(r'\{[\s\S]*\}', result)
+                            if json_match:
+                                sub_results.append(json.loads(json_match.group()))
+                        except Exception as e:
+                            print(f"    ERROR in sub-group {si+1}: {e}")
+                    if sub_results:
+                        merged = {"domain": domain, "consolidated_findings": [], "positive_controls": [], "asvs_statuses": {}, "dedup_log": []}
+                        for sr in sub_results:
+                            merged["consolidated_findings"].extend(sr.get("consolidated_findings", []))
+                            merged["positive_controls"].extend(sr.get("positive_controls", []))
+                            merged["asvs_statuses"].update(sr.get("asvs_statuses", {}))
+                            merged["dedup_log"].extend(sr.get("dedup_log", []))
+                        consolidation_ns.set(domain, merged)
+                        return domain, merged
+                    return domain, None
+
+                try:
+                    result, _ = await call_llm(
+                        provider=FAST_PROVIDER, model=FAST_MODEL,
+                        messages=messages,
+                        parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=600,
+                    )
+                    json_match = re.search(r'\{[\s\S]*\}', result)
+                    if json_match:
+                        consolidated = json.loads(json_match.group())
+                        consolidation_ns.set(domain, consolidated)
+                        print(f"    Result: {len(consolidated.get('consolidated_findings', []))} findings")
+                        return domain, consolidated
+                    return domain, None
+                except Exception as e:
+                    print(f"    ERROR: {e}")
+                    return domain, None
+
+        consolidation_results = await asyncio.gather(*[
+            consolidate_domain(domain, rpts) for domain, rpts in domain_reports.items()
+        ])
+
+        for domain, consolidated in consolidation_results:
+            if consolidated:
+                domain_consolidated[domain] = consolidated
+            else:
+                rpts = domain_reports[domain]
+                fallback = {"domain": domain, "consolidated_findings": [], "positive_controls": [], "asvs_statuses": {}, "dedup_log": []}
+                for report_key, extracted in rpts.items():
+                    for fi, finding in enumerate(extracted.get("findings", [])):
+                        asvs_sec = finding.get("asvs_section", "")
+                        fallback["consolidated_findings"].append({
+                            "temp_id": f"{domain.upper()}-{fi+1}",
+                            "severity": finding.get("severity", "Medium"),
+                            "title": finding.get("title", "Unknown"),
+                            "description": finding.get("description", ""),
+                            "cwe": finding.get("cwe"),
+                            "asvs_sections": [asvs_sec],
+                            "asvs_levels": [get_asvs_level(asvs_sec)],
+                            "affected_files": finding.get("affected_files", []),
+                            "source_reports": [report_key],
+                            "recommended_remediation": finding.get("recommended_remediation", ""),
+                            "merged_from": [],
+                        })
+                    section = extracted.get("asvs_section", "")
+                    if section:
+                        fallback["asvs_statuses"][section] = {
+                            "status": extracted.get("asvs_status", "Unknown"),
+                            "title": extracted.get("asvs_section_title", ""),
+                        }
+                    fallback["positive_controls"].extend(extracted.get("positive_controls", []))
+                domain_consolidated[domain] = fallback
+
+        total_consolidated = sum(len(d.get("consolidated_findings", [])) for d in domain_consolidated.values())
+        print(f"\nTotal consolidated findings (pre-cross-domain): {total_consolidated}")
+
+        # ============================================================
+        # PHASE 3.5: Cross-Domain Deduplication
+        # ============================================================
+        print("\n=== PHASE 3.5: Cross-domain deduplication ===")
+
+        xd_all = []
+        for domain, data in domain_consolidated.items():
+            for fi, finding in enumerate(data.get("consolidated_findings", [])):
+                finding["_xd_domain"] = domain
+                finding["_xd_idx"] = fi
+                xd_all.append(finding)
+
+        def primary_file(finding):
+            af = finding.get("affected_files", [])
+            if not af:
+                return ""
+            first = af[0]
+            if isinstance(first, dict):
+                return first.get("file", "").split(":")[0].split(" (")[0].strip().strip("`")
+            return str(first).split(":")[0].split(" (")[0].strip().strip("`")
+
+        def norm_title(t):
+            t = t.lower().strip()
+            for noise in ["completely ", "critical: ", "high: ", "medium: ", "low: "]:
+                t = t.replace(noise, "")
+            t = re.sub(r'[^a-z0-9 ]', '', t)
+            t = re.sub(r'\s+', ' ', t).strip()
+            return t
+
+        file_groups = {}
+        for finding in xd_all:
+            pf = primary_file(finding)
+            if pf:
+                file_groups.setdefault(pf, []).append(finding)
+
+        xd_merge_count = 0
+        xd_removed = set()
+
+        def merge_into(primary, duplicate):
+            for sr in duplicate.get("source_reports", []):
+                if sr not in primary.get("source_reports", []):
+                    primary.setdefault("source_reports", []).append(sr)
+            for sec in duplicate.get("asvs_sections", []):
+                if sec not in primary.get("asvs_sections", []):
+                    primary.setdefault("asvs_sections", []).append(sec)
+            for lv in duplicate.get("asvs_levels", []):
+                if lv not in primary.get("asvs_levels", []):
+                    primary.setdefault("asvs_levels", []).append(lv)
+            primary.setdefault("merged_from", []).append(duplicate.get("temp_id", "unknown"))
+            if len(duplicate.get("description", "")) > len(primary.get("description", "")):
+                primary["description"] = duplicate["description"]
+            if len(duplicate.get("recommended_remediation", "")) > len(primary.get("recommended_remediation", "")):
+                primary["recommended_remediation"] = duplicate["recommended_remediation"]
+
+        for pf, group in file_groups.items():
+            if len(group) < 2:
+                continue
+            title_clusters = {}
+            for finding in group:
+                nt = norm_title(finding.get("title", ""))
+                title_clusters.setdefault(nt, []).append(finding)
+            for nt, cluster in title_clusters.items():
+                if len(cluster) < 2:
+                    continue
+                cluster.sort(key=lambda f: len(f.get("source_reports", [])), reverse=True)
+                primary = cluster[0]
+                for duplicate in cluster[1:]:
+                    dup_key = (duplicate["_xd_domain"], duplicate["_xd_idx"])
+                    if dup_key in xd_removed:
+                        continue
+                    merge_into(primary, duplicate)
+                    xd_removed.add(dup_key)
+                    xd_merge_count += 1
+
+        # LLM-assisted dedup for large file groups
+        xd_llm_groups = {pf: [f for f in group if (f["_xd_domain"], f["_xd_idx"]) not in xd_removed]
+                         for pf, group in file_groups.items()}
+        xd_llm_groups = {pf: g for pf, g in xd_llm_groups.items() if len(g) >= 3}
+
+        if xd_llm_groups:
+            XD_DEDUP_PROMPT = """You are deduplicating security findings that affect the SAME file but came from DIFFERENT ASVS domain groups.
+
+Two findings are TRUE DUPLICATES if they describe the EXACT SAME bug and a developer fixing one would fix the other.
+
+Return JSON: {"merges": [{"keep": "TEMP-ID", "absorb": ["TEMP-ID-1"], "reason": "same bug"}]}
+If no duplicates: {"merges": []}"""
+
+            for pf, group in xd_llm_groups.items():
+                group_data = [{"temp_id": f"{f['_xd_domain']}:{f.get('temp_id','?')}", "title": f.get("title",""), "description": f.get("description","")[:500], "asvs_sections": f.get("asvs_sections",[]), "affected_files": f.get("affected_files",[])[:3]} for f in group]
+                messages = [{"role": "user", "content": f"{XD_DEDUP_PROMPT}\n\nFile: {pf}\n\n{json.dumps(group_data, indent=2, default=str)}"}]
+                if count_message_tokens(messages, FAST_PROVIDER, FAST_MODEL) > int(FAST_CONTEXT_WINDOW * 0.60):
+                    continue
+                try:
+                    result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL, messages=messages, parameters=FAST_PARAMS, timeout=120)
+                    json_match = re.search(r'\{[\s\S]*\}', result)
+                    if json_match:
+                        dedup_result = json.loads(json_match.group())
+                        lookup = {f"{f['_xd_domain']}:{f.get('temp_id','?')}": f for f in group}
+                        for merge in dedup_result.get("merges", []):
+                            keep = lookup.get(merge.get("keep",""))
+                            if not keep:
+                                continue
+                            for abs_key in merge.get("absorb", []):
+                                abs_f = lookup.get(abs_key)
+                                if abs_f and (abs_f["_xd_domain"], abs_f["_xd_idx"]) not in xd_removed:
+                                    merge_into(keep, abs_f)
+                                    xd_removed.add((abs_f["_xd_domain"], abs_f["_xd_idx"]))
+                                    xd_merge_count += 1
+                except Exception:
+                    pass
+
+        if xd_removed:
+            for domain, data in domain_consolidated.items():
+                data["consolidated_findings"] = [f for fi, f in enumerate(data.get("consolidated_findings", [])) if (domain, fi) not in xd_removed]
+            for domain, data in domain_consolidated.items():
+                for f in data.get("consolidated_findings", []):
+                    f.pop("_xd_domain", None)
+                    f.pop("_xd_idx", None)
+
+        total_after_xd = sum(len(d.get("consolidated_findings", [])) for d in domain_consolidated.values())
+        print(f"Cross-domain dedup: {xd_merge_count} merges, {total_consolidated} → {total_after_xd} findings")
+
+        # ============================================================
+        # PHASE 4: Final Merge and Report Generation
+        # ============================================================
+        print("\n=== PHASE 4: Final merge and report generation ===")
+
+        severity_order = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3, "Informational": 4}
+        all_findings = []
+
+        for domain, data in domain_consolidated.items():
+            for finding in data.get("consolidated_findings", []):
+                finding["_domain"] = domain
+                if "asvs_levels" not in finding or not finding["asvs_levels"]:
+                    levels = set()
+                    for sec in finding.get("asvs_sections", []):
+                        levels.add(get_asvs_level(sec))
+                    finding["asvs_levels"] = sorted(levels) if levels else [level]
+                all_findings.append(finding)
+
+        all_findings.sort(key=lambda f: severity_order.get(f.get("severity", "Informational"), 4))
+
+        for i, finding in enumerate(all_findings, 1):
+            finding["global_id"] = f"FINDING-{i:03d}"
+
+        # Cross-references
+        print("Building cross-references...")
+
+        def extract_primary_file(finding):
+            af = finding.get("affected_files", [])
+            if not af:
+                return ""
+            first = af[0]
+            raw = first.get("file", "") if isinstance(first, dict) else str(first)
+            return re.sub(r'[:\s(].*', '', raw).strip().strip("`")
+
+        def extract_function_names(finding):
+            names = set()
+            for af in finding.get("affected_files", []):
+                raw = af.get("file", "") if isinstance(af, dict) else str(af)
+                for m in re.finditer(r'(?:^|[:\s])([a-zA-Z_]\w*(?:\.\w+)*)\s*\(', raw):
+                    names.add(m.group(1))
+            return names
+
+        by_file, by_cwe, by_func = {}, {}, {}
+        for finding in all_findings:
+            gid = finding["global_id"]
+            pf = extract_primary_file(finding)
+            if pf:
+                by_file.setdefault(pf, set()).add(gid)
+            cwe = finding.get("cwe", "")
+            if cwe and cwe != "null":
+                by_cwe.setdefault(cwe, set()).add(gid)
+            for fn in extract_function_names(finding):
+                by_func.setdefault(fn, set()).add(gid)
+
+        for finding in all_findings:
+            gid = finding["global_id"]
+            related = set()
+            cwe = finding.get("cwe", "")
+            if cwe == "null":
+                cwe = ""
+            if cwe and cwe in by_cwe:
+                related |= by_cwe[cwe]
+            for fn in extract_function_names(finding):
+                if fn in by_func:
+                    related |= by_func[fn]
+            pf = extract_primary_file(finding)
+            if pf and pf in by_file:
+                for oid in by_file[pf]:
+                    if oid == gid:
+                        continue
+                    other = next((f for f in all_findings if f["global_id"] == oid), None)
+                    if not other:
+                        continue
+                    ocwe = other.get("cwe", "")
+                    if ocwe == "null":
+                        ocwe = ""
+                    if (cwe and ocwe and cwe == ocwe) or (extract_function_names(finding) & extract_function_names(other)):
+                        related.add(oid)
+            related.discard(gid)
+            finding["related_findings"] = sorted(related)[:10] if related else []
+
+        all_asvs_statuses = {}
+        for domain, data in domain_consolidated.items():
+            for section, info in data.get("asvs_statuses", {}).items():
+                all_asvs_statuses[section] = info
+        for rk, extracted in all_extracted.items():
+            section = extracted.get("asvs_section", "")
+            if section and section not in all_asvs_statuses:
+                all_asvs_statuses[section] = {
+                    "status": extracted.get("asvs_status", "Unknown"),
+                    "title": extracted.get("asvs_section_title", ""),
+                }
+
+        all_positive_controls = []
+        for domain, data in domain_consolidated.items():
+            for ctrl in data.get("positive_controls", []):
+                ctrl["_domain"] = domain
+                all_positive_controls.append(ctrl)
+
+        severity_counts = {"Critical": 0, "High": 0, "Medium": 0, "Low": 0, "Informational": 0}
+        for f in all_findings:
+            sev = f.get("severity", "Informational")
+            severity_counts[sev] = severity_counts.get(sev, 0) + 1
+
+        commit_info = "N/A"
+        for part in directories[0].split("/"):
+            if len(part) >= 7 and re.match(r'^[0-9a-f]+$', part):
+                commit_info = part
+                break
+
+        findings_by_severity = {}
+        for f in all_findings:
+            findings_by_severity.setdefault(f.get("severity", "Informational"), []).append(f)
+
+        # Executive summary (Opus)
+        print(f"Generating executive summary...")
+        exec_summary_prompt = f"""Generate the opening sections of a security audit consolidated report in Markdown.
+
+Audit scope: up to {level}
+
+Repository: {owner}/{repo}
+Directories: {', '.join(directories)}
+ASVS Level: {level}
+Commit: {commit_info}
+Date: {audit_date}
+Auditor: Tooling Agents
+Source reports: {total_reports}
+Total findings: {len(all_findings)}
+
+Severity: Critical={severity_counts.get('Critical',0)}, High={severity_counts.get('High',0)}, Medium={severity_counts.get('Medium',0)}, Low={severity_counts.get('Low',0)}, Info={severity_counts.get('Informational',0)}
+
+Finding titles:
+"""
+        for f in all_findings:
+            lvs = ", ".join(f.get("asvs_levels", []))
+            exec_summary_prompt += f"- [{f.get('severity')}] [{lvs}] {f.get('global_id')}: {f.get('title')} (ASVS: {', '.join(f.get('asvs_sections', []))})\n"
+
+        exec_summary_prompt += f"""
+Positive controls: {json.dumps(all_positive_controls[:50], indent=2, default=str)}
+
+Generate ONLY:
+1. Report Metadata table (Repository, ASVS Level, Commit, Date, Auditor, Source Reports, Total Findings)
+2. Executive Summary with severity distribution, level coverage, top 5 risks, positive controls
+
+End with ---."""
+
+        exec_result, _ = await call_llm(provider=HEAVY_PROVIDER, model=HEAVY_MODEL,
+            messages=[{"role": "user", "content": exec_summary_prompt}],
+            parameters={**HEAVY_PARAMS, "max_tokens": 32000}, timeout=900)
+        exec_result = sanitize_md_html(exec_result)
+
+        # Findings (Sonnet, batched)
+        FINDING_FORMAT = """For each finding: #### FINDING-NNN: Title, attribute table (Severity, ASVS Level(s), CWE, ASVS sections, Files, Source Reports, Related), Description, Remediation. Use emojis: 🔴 Critical, 🟠 High, 🟡 Medium, 🔵 Low, ⚪ Info. Separate with ---."""
+
+        findings_md_parts = []
+        MAX_PER_BATCH = 30
+        sev_section_names = {"Critical": "3.1 Critical", "High": "3.2 High", "Medium": "3.3 Medium", "Low": "3.4 Low", "Informational": "3.5 Informational"}
+
+        for sev in ["Critical", "High", "Medium", "Low", "Informational"]:
+            sev_findings = findings_by_severity.get(sev, [])
+            if not sev_findings:
+                continue
+            for sb_idx in range(0, len(sev_findings), MAX_PER_BATCH):
+                batch = sev_findings[sb_idx:sb_idx + MAX_PER_BATCH]
+                is_first = (sb_idx == 0)
+                prompt = f"""Format these {len(batch)} {sev} findings into Markdown.\n{FINDING_FORMAT}\n{"Start with: ### " + sev_section_names[sev] if is_first else "Continue. No header."}\n\n{json.dumps(batch, indent=2, default=str)}\n\nGenerate ALL {len(batch)} findings with full detail."""
+                for attempt in range(3):
+                    try:
+                        result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL,
+                            messages=[{"role": "user", "content": prompt}],
+                            parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=900)
+                        findings_md_parts.append(sanitize_md_html(result))
+                        break
+                    except Exception as e:
+                        if attempt < 2:
+                            await asyncio.sleep(5)
+
+        # Tail sections (Sonnet)
+        tail_prompt = f"""Generate final sections:\n## 4. Positive Security Controls (table)\n## 5. ASVS Compliance Summary (table with status)\n## 6. Cross-Reference Matrix\n\nPositive controls: {json.dumps(all_positive_controls[:100], indent=2, default=str)}\nASVS statuses: {json.dumps(all_asvs_statuses, indent=2, default=str)}\nFindings: {json.dumps([{"id": f["global_id"], "sev": f["severity"], "title": f["title"], "asvs": f.get("asvs_sections",[])} for f in all_findings], indent=2, d [...]
+
+        try:
+            tail_result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL,
+                messages=[{"role": "user", "content": tail_prompt}],
+                parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=900)
+            tail_result = sanitize_md_html(tail_result)
+        except:
+            tail_result = "\n*Tail sections generation failed.*\n"
+
+        # Assemble
+        consolidated_md = exec_result.rstrip() + "\n\n## 3. Findings\n\n" + "\n\n".join(findings_md_parts) + "\n\n---\n\n" + tail_result
+
+        # Section 7: Level Coverage (deterministic)
+        s7 = ["\n\n## 7. Level Coverage Analysis\n", f"\n**Audit scope:** up to {level}\n"]
+        s7.append("| Level | Sections Audited | Findings Found |")
+        s7.append("|-------|-----------------|----------------|")
+        for lv_name, lv_num in [("L1", 1), ("L2", 2), ("L3", 3)]:
+            if lv_num > max_level_num:
+                continue
+            lv_sections = sum(1 for rk, ext in all_extracted.items() if get_asvs_level(ext.get("asvs_section", "")) == lv_name)
+            lv_findings = [f for f in all_findings if lv_name in f.get("asvs_levels", [])]
+            s7.append(f"| {lv_name} | {lv_sections} | {len(lv_findings)} |")
+        s7.append(f"\n**Total consolidated findings: {len(all_findings)}**")
+        s7.append("\n*End of Consolidated Security Audit Report*")
+        consolidated_md += "\n".join(s7)
+
+        # Issues file
+        informational_ids = set(f["global_id"] for f in all_findings if "informational" in f.get("severity","").lower())
+        actionable = [f for f in all_findings if f["global_id"] not in informational_ids]
+
+        ISSUE_FMT = """---\n## Issue: FINDING-NNN - Title\n**Labels:** bug, security, priority:sev\n**Description:**\n### Summary\n### Details\n### Remediation\n### Acceptance Criteria\n- [ ] Fixed\n- [ ] Test added\n### References\n### Priority"""
+
+        issues_parts = []
+        for bi in range(0, len(actionable), 75):
+            batch = actionable[bi:bi+75]
+            is_first = (bi == 0)
+            prompt = f"""Generate GitHub issues for {len(batch)} findings.\n{ISSUE_FMT}\n{"Start with: # Security Issues" if is_first else "Continue. No header."}\n\n{json.dumps(batch, indent=2, default=str)}"""
+            for attempt in range(3):
+                try:
+                    result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL,
+                        messages=[{"role": "user", "content": prompt}],
+                        parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=900)
+                    issues_parts.append(sanitize_md_html(result))
+                    break
+                except:
+                    if attempt < 2:
+                        await asyncio.sleep(5)
+                    else:
+                        # Fallback
+                        fb = []
+                        for f in batch:
+                            fb.append(f"---\n\n## Issue: {f['global_id']} - {sanitize_md_html(f.get('title',''))}\n\n**Labels:** bug, security, priority:{f.get('severity','Medium').lower()}\n\n**Description:**\n\n{sanitize_md_html(f.get('description',''))}\n\n**Remediation:** {sanitize_md_html(f.get('recommended_remediation',''))}\n\n**Priority:** {f.get('severity','Medium')}")
+                        issues_parts.append("\n\n".join(fb))
+
+        issues_md = "\n\n".join(issues_parts)
+
+        # Quality checks
+        print(f"\n=== Quality Checks ===")
+        finding_sections = set(re.findall(r'#### FINDING-(\d{3})', consolidated_md))
+        print(f"Finding sections: {len(finding_sections)} (expected: {len(all_findings)})")
+        issue_headers = set(re.findall(r'## Issue: (FINDING-\d{3})', issues_md))
+        print(f"Issues: {len(issue_headers)} (expected: {len(actionable)})")
+
+        # Push to GitHub
+        print("\n=== Pushing files ===")
+
+        async def push_file(path, content_str, message):
+            existing_sha = None
+            check = await http_client.get(f"{GITHUB_API}/repos/{owner}/{repo}/contents/{path}", headers=headers, params={"ref": default_branch})
+            if check.status_code == 200:
+                existing_sha = check.json().get("sha")
+            payload = {"message": message, "content": base64.b64encode(content_str.encode("utf-8")).decode("ascii"), "branch": default_branch}
+            if existing_sha:
+                payload["sha"] = existing_sha
+            resp = await http_client.put(f"{GITHUB_API}/repos/{owner}/{repo}/contents/{path}", headers=headers, json=payload)
+            print(f"  {'OK' if resp.status_code in (200,201) else 'ERROR'}: {path}")
+
+        await push_file(f"{output_directory}/{consolidated_filename}", consolidated_md, f"Add consolidated audit report ({level})")
+        await push_file(f"{output_directory}/{issues_filename}", issues_md, f"Add security issues ({level})")
+
+        print(f"\n=== Done ===")
+        print(f"Total findings: {len(all_findings)}, Actionable issues: {len(actionable)}")
+
+        return {
+            "outputText": f"Consolidated report and issues pushed.\n"
+                          f"Repository: {owner}/{repo}\n"
+                          f"Level: {level}\n"
+                          f"Findings: {len(all_findings)}\n"
+                          f"Files: {output_directory}/{consolidated_filename}, {output_directory}/{issues_filename}"
+        }
+    finally:
+        await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/discover_codebase_architecture.py b/repos/apache/ASVS/agents/discover_codebase_architecture.py
new file mode 100644
index 0000000..6c48b5d
--- /dev/null
+++ b/repos/apache/ASVS/agents/discover_codebase_architecture.py
@@ -0,0 +1,366 @@
+# discover_codebase_architecture
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+    mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+    http_client = httpx.AsyncClient()
+    try:
+        import json
+        import re
+
+        input_namespace = input_dict.get("inputNamespace", "")
+
+        namespaces = [ns.strip() for ns in input_namespace.split(",") if ns.strip()]
+        if not namespaces:
+            all_ns = data_store.list_namespaces()
+            file_ns = [ns for ns in all_ns if ns.startswith("files:")]
+            if file_ns:
+                namespaces = file_ns
+
+        if not namespaces:
+            return {"outputText": json.dumps({"error": f"No namespaces provided. Available: {data_store.list_namespaces()}"})}
+
+        repo_name = "unknown"
+        for ns in namespaces:
+            if ns.startswith("files:"):
+                repo_name = ns.replace("files:", "")
+                break
+
+        print(f"Discovering architecture for: {repo_name}", flush=True)
+        print(f"Namespaces: {namespaces}", flush=True)
+
+        # =============================================================
+        # Model configuration
+        # =============================================================
+        PROVIDER = "bedrock"
+        MODEL = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+        PARAMS = {"temperature": 0.7, "max_tokens": 16384}
+        CONTEXT_WINDOW = get_context_window(PROVIDER, MODEL)
+        SAFE_LIMIT = int(CONTEXT_WINDOW * 0.40)
+
+        # =============================================================
+        # Step 1: Read file paths and previews
+        # =============================================================
+        print("\n=== Step 1: Reading file paths and previews ===", flush=True)
+
+        SKIP_DIRS = {
+            'node_modules', 'vendor', 'third_party', 'third-party',
+            'dist', 'build', 'out', 'target',
+            '__pycache__', '.pytest_cache', '.mypy_cache', 'coverage', '.next', '.nuxt',
+            'assets', 'images', 'img', 'static/images', 'static/fonts', 'static/webfonts',
+            'public/images', 'fonts', 'webfonts',
+            '.github/workflows',
+            'venv', '.venv', 'env', '.env',
+            '.git', '.idea', '.vscode',
+        }
+        SKIP_FILES = {
+            'package-lock.json', 'yarn.lock', 'poetry.lock', 'Cargo.lock',
+            'composer.lock', 'pnpm-lock.yaml', 'Gemfile.lock', 'uv.lock',
+            'LICENSE', 'LICENSE.md', 'LICENSE.txt',
+            'README.md', 'README.rst', 'README.txt', 'README',
+            'CHANGELOG.md', 'CHANGELOG', 'CONTRIBUTING.md', 'CODE_OF_CONDUCT.md',
+            '.gitignore', '.dockerignore', '.prettierrc', '.eslintrc', '.editorconfig',
+            '.npmrc', '.yarnrc',
+        }
+        SKIP_EXTENSIONS = {
+            '.min.js', '.min.css', '.bundle.js', '.bundle.css',
+            '.map', '.css', '.scss', '.less', '.sass', '.styl',
+            '.woff', '.woff2', '.ttf', '.eot', '.otf',
+            '.png', '.jpg', '.jpeg', '.gif', '.svg', '.ico', '.webp', '.bmp',
+            '.mp3', '.mp4', '.wav', '.avi', '.mov', '.webm', '.ogg',
+            '.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx',
+            '.zip', '.tar', '.gz', '.rar', '.7z', '.lock',
+            '.exe', '.dll', '.so', '.dylib', '.pyc', '.pyo', '.class', '.o', '.obj',
+        }
+
+        def should_skip_file(filepath):
+            path_lower = filepath.lower()
+            parts = filepath.split('/')
+            for part in parts[:-1]:
+                if part.lower() in SKIP_DIRS:
+                    return True
+            filename = parts[-1] if parts else ''
+            if filename in SKIP_FILES:
+                return True
+            for ext in SKIP_EXTENSIONS:
+                if path_lower.endswith(ext):
+                    return True
+            return False
+
+        all_files = {}
+        for ns in namespaces:
+            ns_store = data_store.use_namespace(ns)
+            keys = ns_store.list_keys()
+            file_contents = ns_store.get_many(keys) if keys else {}
+            for k, v in file_contents.items():
+                if v is not None and not should_skip_file(k):
+                    content = v if isinstance(v, str) else json.dumps(v, default=str) if v else ""
+                    all_files[k] = content
+
+        print(f"Total code files: {len(all_files)}", flush=True)
+
+        if not all_files:
+            return {"outputText": json.dumps({"error": f"No code files found in namespaces {namespaces}"})}
+
+        file_previews = {}
+        for path, content in all_files.items():
+            lines = content.split('\n')
+            file_previews[path] = '\n'.join(lines[:30])
+
+        total_lines = sum(len(content.split('\n')) for content in all_files.values())
+        print(f"Total lines: {total_lines}", flush=True)
+
+        # =============================================================
+        # Step 2: Architecture classification (Sonnet, batched)
+        # =============================================================
+        print("\n=== Step 2: Architecture classification ===", flush=True)
+
+        CLASSIFY_PROMPT = """You are a security architect analyzing a codebase structure.
+
+Given the file paths and code previews below, identify the codebase's security architecture.
+
+Return ONLY a JSON object with this structure:
+{
+  "framework": "e.g., FastAPI, Django, Flask, Express, Spring",
+  "language": "e.g., Python, Java, JavaScript",
+  "auth_systems": [
+    {"name": "description", "files": ["path1", "path2"]}
+  ],
+  "api_layers": [
+    {"name": "description", "files": ["path1", "path2"]}
+  ],
+  "data_layer": {
+    "database": "e.g., SQLAlchemy, Django ORM",
+    "encryption": "e.g., Fernet, bcrypt",
+    "secrets": "e.g., Vault, env vars, config file",
+    "files": ["path1", "path2"]
+  },
+  "execution_model": {
+    "description": "e.g., async web server, task workers, DAG executor",
+    "files": ["path1", "path2"]
+  },
+  "security_relevant_areas": [
+    {"area": "short name", "description": "what it does", "files": ["path1", "path2"]}
+  ],
+  "trust_model": "one paragraph describing who is trusted and what boundaries exist"
+}
+"""
+        template_tokens = count_tokens(CLASSIFY_PROMPT, PROVIDER, MODEL)
+        preview_budget = SAFE_LIMIT - template_tokens
+
+        path_list = "\n".join(sorted(all_files.keys()))
+        path_tokens = count_tokens(path_list, PROVIDER, MODEL)
+
+        if path_tokens + template_tokens < SAFE_LIMIT:
+            entries = []
+            current_tokens = 0
+            for path in sorted(all_files.keys()):
+                entry = f"\n--- {path} ---\n{file_previews[path]}\n"
+                entry_tokens = count_tokens(entry, PROVIDER, MODEL)
+                if current_tokens + entry_tokens > preview_budget:
+                    break
+                entries.append(entry)
+                current_tokens += entry_tokens
+            classify_content = CLASSIFY_PROMPT + "\nFILES:\n" + "".join(entries)
+        else:
+            classify_content = CLASSIFY_PROMPT + f"\nFILE PATHS ({len(all_files)} files):\n" + path_list
+
+        print(f"  Classification prompt: {count_tokens(classify_content, PROVIDER, MODEL)} tokens", flush=True)
+
+        architecture = {}
+        for attempt in range(2):
+            try:
+                result, _ = await call_llm(
+                    provider=PROVIDER, model=MODEL,
+                    messages=[{"role": "user", "content": classify_content}],
+                    parameters=PARAMS,
+                    timeout=300,
+                )
+                json_match = re.search(r'\{[\s\S]*\}', result)
+                if json_match:
+                    architecture = json.loads(json_match.group())
+                    print(f"  Framework: {architecture.get('framework', '?')}", flush=True)
+                    print(f"  Auth systems: {len(architecture.get('auth_systems', []))}", flush=True)
+                    print(f"  Security areas: {len(architecture.get('security_relevant_areas', []))}", flush=True)
+                    break
+            except Exception as e:
+                if attempt == 0:
+                    print(f"  Attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+                    await asyncio.sleep(5)
+                else:
+                    print(f"  Classification FAILED: {e}", flush=True)
+
+        if not architecture:
+            return {"outputText": json.dumps({"error": "Failed to classify codebase architecture"})}
+
+        # =============================================================
+        # Step 3: Generate security domains (Sonnet)
+        # =============================================================
+        print("\n=== Step 3: Generating security domains ===", flush=True)
+
+        asvs_sections_available = []
+        try:
+            asvs_ns = data_store.use_namespace("asvs")
+            all_keys = asvs_ns.list_keys()
+            req_keys = [k for k in all_keys if k.startswith("asvs:requirements:")]
+            for rk in sorted(req_keys):
+                req = asvs_ns.get(rk)
+                if req:
+                    section_id = rk.replace("asvs:requirements:", "")
+                    req_level = req.get("level", "?")
+                    desc = req.get("req_description", "")[:100]
+                    asvs_sections_available.append(f"{section_id} (L{req_level}): {desc}")
+        except Exception as e:
+            print(f"  WARNING: Could not load ASVS sections: {e}", flush=True)
+
+        asvs_list = "\n".join(asvs_sections_available[:200])
+
+        DOMAIN_PROMPT = f"""Based on this codebase architecture, generate security audit domains.
+
+## Codebase Architecture
+{json.dumps(architecture, indent=2, default=str)}
+
+## Available ASVS Sections
+{asvs_list}
+
+## Instructions
+
+Generate security domains that reflect THIS codebase's actual architecture.
+Each domain should:
+1. Have a short snake_case name
+2. Map to specific ASVS sections from the list above
+3. List the files that belong to it (use exact paths from the architecture data)
+4. Include a context paragraph that an auditor needs to understand this domain
+
+Group ASVS sections by the code area they'd be testing, NOT by ASVS chapter number.
+Every ASVS section should appear in exactly one domain.
+
+Return ONLY a JSON object:
+{{
+  "domains": [
+    {{
+      "name": "snake_case_name",
+      "description": "what this domain covers",
+      "asvs_sections": ["X.Y.Z", ...],
+      "files": ["path/to/file.py", ...],
+      "context": "paragraph explaining the architecture of this domain for auditors"
+    }}
+  ]
+}}"""
+
+        domains = []
+        for attempt in range(2):
+            try:
+                result, _ = await call_llm(
+                    provider=PROVIDER, model=MODEL,
+                    messages=[{"role": "user", "content": DOMAIN_PROMPT}],
+                    parameters={**PARAMS, "max_tokens": 32000},
+                    timeout=300,
+                )
+                json_match = re.search(r'\{[\s\S]*\}', result)
+                if json_match:
+                    domain_result = json.loads(json_match.group())
+                    domains = domain_result.get("domains", [])
+                    print(f"  Generated {len(domains)} domains", flush=True)
+                    for d in domains:
+                        print(f"    {d['name']}: {len(d.get('asvs_sections', []))} sections, {len(d.get('files', []))} files", flush=True)
+                    break
+            except Exception as e:
+                if attempt == 0:
+                    print(f"  Attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+                    await asyncio.sleep(5)
+                else:
+                    print(f"  Domain generation FAILED: {e}", flush=True)
+
+        if not domains:
+            return {"outputText": json.dumps({"error": "Failed to generate security domains"})}
+
+        for domain in domains:
+            line_count = 0
+            for path in domain.get("files", []):
+                if path in all_files:
+                    line_count += len(all_files[path].split('\n'))
+            domain["estimated_lines"] = line_count
+
+        # =============================================================
+        # Step 4: Generate false positive guidance (Sonnet)
+        # =============================================================
+        print("\n=== Step 4: Generating false positive guidance ===", flush=True)
+
+        FP_PROMPT = f"""Based on this codebase architecture, identify patterns that an ASVS security auditor would INCORRECTLY flag as vulnerabilities.
+
+## Codebase Architecture
+{json.dumps(architecture, indent=2, default=str)}
+
+For each false positive pattern, write a concise statement:
+"[What auditor would flag] is intentional because [reason] — auditors should focus on [what matters instead]"
+
+Return ONLY a JSON array of strings:
+[
+  "Pattern X is intentional because Y — auditors should focus on Z instead",
+  ...
+]"""
+
+        false_positive_guidance = []
+        try:
+            result, _ = await call_llm(
+                provider=PROVIDER, model=MODEL,
+                messages=[{"role": "user", "content": FP_PROMPT}],
+                parameters=PARAMS,
+                timeout=120,
+            )
+            json_match = re.search(r'\[[\s\S]*\]', result)
+            if json_match:
+                false_positive_guidance = json.loads(json_match.group())
+                print(f"  Generated {len(false_positive_guidance)} patterns", flush=True)
+        except Exception as e:
+            print(f"  False positive generation failed ({type(e).__name__}), continuing without", flush=True)
+
+        # =============================================================
+        # Step 5: Assemble output
+        # =============================================================
+        domain_groups = {}
+        for domain in domains:
+            domain_groups[domain["name"]] = domain.get("asvs_sections", [])
+
+        passes = []
+        for domain in domains:
+            passes.append({
+                "name": domain["name"],
+                "description": domain.get("description", ""),
+                "asvs_sections": domain.get("asvs_sections", []),
+                "files": domain.get("files", []),
+                "domain_context": domain.get("context", ""),
+                "estimated_lines": domain.get("estimated_lines", 0),
+            })
+        passes.sort(key=lambda p: p["estimated_lines"])
+
+        pass_config = {
+            "repository": repo_name,
+            "architecture_summary": architecture.get("trust_model", ""),
+            "framework": architecture.get("framework", ""),
+            "language": architecture.get("language", ""),
+            "total_files": len(all_files),
+            "total_lines": total_lines,
+            "passes": passes,
+            "domain_groups": domain_groups,
+            "false_positive_guidance": false_positive_guidance,
+        }
+
+        discovery_ns = data_store.use_namespace(f"discovery:{repo_name}")
+        discovery_ns.set("pass_config", pass_config)
+        discovery_ns.set("architecture", architecture)
+
+        print(f"\n=== Discovery complete ===", flush=True)
+        print(f"Repository: {repo_name}", flush=True)
+        print(f"Domains: {len(domains)}", flush=True)
+        for p in passes:
+            print(f"  {p['name']}: {len(p['asvs_sections'])} sections, {p['estimated_lines']} lines", flush=True)
+
+        return {"outputText": json.dumps(pass_config, indent=2, default=str)}
+
+    finally:
+        await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/download_github_repo_to_data_store.py b/repos/apache/ASVS/agents/download_github_repo_to_data_store.py
new file mode 100644
index 0000000..fa3576f
--- /dev/null
+++ b/repos/apache/ASVS/agents/download_github_repo_to_data_store.py
@@ -0,0 +1,149 @@
+# download_github_repo_to_datastore
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+    mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+    http_client = httpx.AsyncClient()
+    try:
+        input_text = input_dict.get("inputText", "")
+        lines = input_text.strip().split("\n")
+
+        repo = None
+        token = None
+        path_prefix = ""
+
+        for line in lines:
+            line = line.strip()
+            if not line:
+                continue
+            if line.startswith("ghp_") or line.startswith("github_pat_") or (len(line) > 30 and line.isalnum()):
+                token = line
+            elif "/" in line and repo is None:
+                raw = line.split()[0].strip("/")
+                parts = raw.split("/")
+                if len(parts) >= 2:
+                    repo = f"{parts[0]}/{parts[1]}"
+                    if len(parts) > 2:
+                        path_prefix = "/".join(parts[2:])
+
+        if not repo:
+            return {"outputText": "Error: Could not find a valid repo in format 'owner/repo' in the input."}
+
+        headers = {
+            "Accept": "application/vnd.github+json",
+            "X-GitHub-Api-Version": "2022-11-28"
+        }
+        if token:
+            headers["Authorization"] = f"Bearer {token}"
+
+        repo_url = f"https://api.github.com/repos/{repo}"
+        response = await http_client.get(repo_url, headers=headers)
+
+        if response.status_code != 200:
+            return {"outputText": f"Error fetching repo info: {response.status_code} - repo='{repo}' token_present={token is not None} - {response.text}"}
+
+        repo_info = response.json()
+        default_branch = repo_info.get("default_branch", "main")
+
+        tree_url = f"https://api.github.com/repos/{repo}/git/trees/{default_branch}?recursive=1"
+        response = await http_client.get(tree_url, headers=headers)
+
+        if response.status_code != 200:
+            return {"outputText": f"Error fetching repo tree: {response.status_code} - {response.text}"}
+
+        tree_data = response.json()
+        tree_items = tree_data.get("tree", [])
+        files_to_fetch = [item for item in tree_items if item.get("type") == "blob"]
+
+        # Filter by path prefix if provided
+        if path_prefix:
+            files_to_fetch = [item for item in files_to_fetch if item.get("path", "").startswith(path_prefix + "/") or item.get("path", "") == path_prefix]
+            print(f"Path prefix '{path_prefix}': {len(files_to_fetch)} files match", flush=True)
+
+        files_ns = data_store.use_namespace(f"files:{repo}")
+
+        fetched_count = 0
+        skipped_binary = 0
+        skipped_vendor = 0
+        skipped_large = 0
+        error_count = 0
+        errors = []
+
+        VENDOR_DIRS = {
+            'node_modules', 'vendor', 'third_party', 'third-party',
+            '.git',
+        }
+
+        MAX_FILE_SIZE = 1_000_000
+
+        import base64
+
+        for item in files_to_fetch:
+            file_path = item.get("path", "")
+            file_size = item.get("size", 0)
+
+            parts = file_path.split("/")
+            if any(part.lower() in VENDOR_DIRS for part in parts[:-1]):
+                skipped_vendor += 1
+                continue
+
+            if file_size > MAX_FILE_SIZE:
+                skipped_large += 1
+                continue
+
+            try:
+                content_url = f"https://api.github.com/repos/{repo}/contents/{file_path}?ref={default_branch}"
+                file_response = await http_client.get(content_url, headers=headers)
+
+                if file_response.status_code == 200:
+                    file_data = file_response.json()
+                    content_b64 = file_data.get("content", "")
+
+                    if content_b64:
+                        try:
+                            content = base64.b64decode(content_b64).decode("utf-8")
+                            files_ns.set(file_path, content)
+                            fetched_count += 1
+                        except UnicodeDecodeError:
+                            skipped_binary += 1
+                    else:
+                        skipped_binary += 1
+                else:
+                    error_count += 1
+                    if len(errors) < 10:
+                        errors.append(f"{file_path}: {file_response.status_code}")
+            except Exception as e:
+                error_count += 1
+                if len(errors) < 10:
+                    errors.append(f"{file_path}: {str(e)}")
+
+        summary_lines = [
+            f"Repository: {repo}",
+            f"Default branch: {default_branch}",
+        ]
+        if path_prefix:
+            summary_lines.append(f"Path prefix: {path_prefix}")
+        summary_lines += [
+            f"Total files in scope: {len(files_to_fetch)}",
+            f"Files fetched and stored: {fetched_count}",
+            f"Skipped (binary): {skipped_binary}",
+            f"Skipped (vendor/third-party dirs): {skipped_vendor}",
+            f"Skipped (>1MB): {skipped_large}",
+            f"Errors: {error_count}",
+            f"",
+            f"Data stored in namespace: files:{repo}",
+        ]
+
+        if errors:
+            summary_lines.append("")
+            summary_lines.append("First few errors:")
+            for err in errors:
+                summary_lines.append(f"  - {err}")
+
+        return {"outputText": "\n".join(summary_lines)}
+
+    finally:
+        await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/orchestrate_asvs_audit_to_github.py b/repos/apache/ASVS/agents/orchestrate_asvs_audit_to_github.py
new file mode 100644
index 0000000..52ed339
--- /dev/null
+++ b/repos/apache/ASVS/agents/orchestrate_asvs_audit_to_github.py
@@ -0,0 +1,368 @@
+# orchestrate_asvs_audit_to_github
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+    mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+    http_client = httpx.AsyncClient()
+    try:
+        import json
+
+        # =============================================================
+        # Parse inputs
+        # =============================================================
+        source_repo = input_dict.get("sourceRepo", "")       # e.g., "apache/airflow" or "apache/airflow/airflow-core/src"
+        source_token = input_dict.get("sourceToken", "")      # PAT for private source repos
+        supplemental_data = input_dict.get("supplementalData", "")  # extra namespaces, comma-separated
+        output_repo = input_dict.get("outputRepo", "")
+        output_token = input_dict.get("outputToken", "")
+        output_directory = input_dict.get("outputDirectory", "")
+        discover = input_dict.get("discover", "true")
+        severity_threshold = input_dict.get("severityThreshold", "")
+        consolidate = input_dict.get("consolidate", "true")
+        level = input_dict.get("level", "")
+
+        if isinstance(discover, str):
+            discover = discover.lower() in ("true", "1", "yes")
+        if isinstance(consolidate, str):
+            consolidate = consolidate.lower() in ("true", "1", "yes")
+
+        level = level.strip().upper()
+        if level and not level.startswith("L"):
+            level = f"L{level}"
+        LEVEL_ORDER = {"L1": 1, "L2": 2, "L3": 3}
+        max_level_num = LEVEL_ORDER.get(level, 3)
+
+        if not source_repo:
+            return {"outputText": "Error: sourceRepo is required (e.g., 'apache/airflow', 'apache/airflow/src', or 'https://github.com/apache/airflow/tree/main/src')"}
+
+        # Derive repo name, path prefix, and namespace from sourceRepo
+        # Accepts:
+        #   "apache/steve"                                    → repo=apache/steve, path=""
+        #   "apache/steve/v3"                                 → repo=apache/steve, path="v3"
+        #   "apache/airflow/airflow-core/src"                 → repo=apache/airflow, path="airflow-core/src"
+        #   "https://github.com/apache/steve/tree/trunk/v3"   → repo=apache/steve, path="v3"
+        import re as _re
+        _source = source_repo.strip().strip("/")
+
+        # Strip GitHub URL prefix and tree/branch segment if present
+        _gh_match = _re.match(
+            r'(?:https?://)?github\.com/([^/]+)/([^/]+?)(?:\.git)?(?:/tree/[^/]+(?:/(.+))?)?$',
+            _source,
+        )
+        if _gh_match:
+            repo_owner_name = f"{_gh_match.group(1)}/{_gh_match.group(2)}"
+            repo_short_name = _gh_match.group(2)
+            source_path_prefix = _gh_match.group(3) or ""
+        else:
+            source_parts = _source.split("/")
+            if len(source_parts) < 2:
+                return {"outputText": f"Error: sourceRepo must be owner/repo format, got '{source_repo}'"}
+            repo_owner_name = f"{source_parts[0]}/{source_parts[1]}"
+            repo_short_name = source_parts[1]
+            source_path_prefix = "/".join(source_parts[2:]) if len(source_parts) > 2 else ""
+
+        code_namespace = f"files:{repo_owner_name}"
+
+        # Reconstruct the download input: owner/repo/path (short form for the download agent)
+        download_source = repo_owner_name
+        if source_path_prefix:
+            download_source += f"/{source_path_prefix}"
+
+        # Build namespace list: code namespace + any supplemental
+        namespaces = [code_namespace]
+        if supplemental_data:
+            for ns in supplemental_data.split(","):
+                ns = ns.strip()
+                if ns and ns not in namespaces:
+                    namespaces.append(ns)
+
+        # Fetch latest commit hash and append repo/hash to output directory
+        source_headers = {"Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28"}
+        if source_token:
+            source_headers["Authorization"] = f"Bearer {source_token}"
+        try:
+            commits_resp = await http_client.get(
+                f"https://api.github.com/repos/{repo_owner_name}/commits?per_page=1",
+                headers=source_headers,
+            )
+            commits_data = commits_resp.json()
+            commit_hash = commits_data[0]["sha"][:7]
+        except Exception as e:
+            print(f"  WARNING: Could not fetch commit hash ({e}), using 'latest'", flush=True)
+            commit_hash = "latest"
+
+        output_directory = f"{output_directory.strip('/')}/{repo_short_name}/{commit_hash}"
+        print(f"  Output directory: {output_directory}", flush=True)
+
+        all_outputs = []
+        successes = []
+        failures = []
+        report_directories = []
+
+        # =============================================================
+        # Helper: filter ASVS sections by level
+        # =============================================================
+        asvs_level_cache = {}
+
+        def load_asvs_levels():
+            if asvs_level_cache:
+                return
+            try:
+                asvs_ns = data_store.use_namespace("asvs")
+                all_keys = asvs_ns.list_keys()
+                req_keys = [k for k in all_keys if k.startswith("asvs:requirements:")]
+                for rk in req_keys:
+                    req = asvs_ns.get(rk)
+                    if req:
+                        section_id = rk.replace("asvs:requirements:", "")
+                        asvs_level_cache[section_id] = int(req.get("level", 1))
+                print(f"  Loaded ASVS levels for {len(asvs_level_cache)} sections", flush=True)
+            except Exception as e:
+                print(f"  WARNING: Could not load ASVS levels: {e}", flush=True)
+
+        def filter_sections_by_level(sections):
+            if not level:
+                return sections
+            load_asvs_levels()
+            return [s for s in sections if asvs_level_cache.get(s, 1) <= max_level_num]
+
+        # =============================================================
+        # Step 1: Download source code
+        # =============================================================
+        print(f"{'='*60}", flush=True)
+        print(f"Step 1: Downloading source code", flush=True)
+        print(f"  Source: {source_repo}", flush=True)
+        print(f"  Repo: {repo_owner_name}", flush=True)
+        if source_path_prefix:
+            print(f"  Path: {source_path_prefix}", flush=True)
+        print(f"  Namespace: {code_namespace}", flush=True)
+        print(f"{'='*60}", flush=True)
+
+        download_input = download_source
+        if source_token:
+            download_input += f"\n{source_token}"
+
+        try:
+            download_result = await gofannon_client.call(
+                agent_name="download_github_repo_to_datastore",
+                input_dict={
+                    "inputText": download_input,
+                }
+            )
+            download_output = download_result.get("outputText", "")
+            print(f"  {download_output}", flush=True)
+        except Exception as e:
+            return {"outputText": f"Download failed: {e}"}
+
+        # =============================================================
+        # Step 2: Discover architecture
+        # =============================================================
+        if discover:
+            print(f"\n{'='*60}", flush=True)
+            print(f"Step 2: Discovering codebase architecture", flush=True)
+            print(f"  Namespaces: {namespaces}", flush=True)
+            print(f"{'='*60}", flush=True)
+
+            try:
+                discovery_result = await gofannon_client.call(
+                    agent_name="discover_codebase_architecture",
+                    input_dict={
+                        "inputNamespace": ",".join(namespaces),
+                    }
+                )
+                pass_config = json.loads(discovery_result.get("outputText", "{}"))
+                if "error" in pass_config:
+                    return {"outputText": f"Discovery failed: {pass_config['error']}"}
+
+                passes = pass_config.get("passes", [])
+                false_positive_guidance = pass_config.get("false_positive_guidance", [])
+                domain_groups = pass_config.get("domain_groups", {})
+                print(f"  Discovery complete: {len(passes)} passes", flush=True)
+            except Exception as e:
+                return {"outputText": f"Discovery agent failed: {e}"}
+
+            # Filter by level
+            for pass_def in passes:
+                pass_def["asvs_sections"] = filter_sections_by_level(pass_def.get("asvs_sections", []))
+            passes = [p for p in passes if p.get("asvs_sections")]
+
+            if level:
+                for dn in list(domain_groups.keys()):
+                    domain_groups[dn] = filter_sections_by_level(domain_groups[dn])
+                domain_groups = {k: v for k, v in domain_groups.items() if v}
+
+            total_sections = sum(len(p.get("asvs_sections", [])) for p in passes)
+            print(f"  After level filter ({level or 'all'}): {total_sections} sections, {len(passes)} passes", flush=True)
+
+            if total_sections == 0:
+                return {"outputText": f"No ASVS sections match level {level}."}
+
+            # =============================================================
+            # Step 3: Audit + push
+            # =============================================================
+            print(f"\n{'='*60}", flush=True)
+            print(f"Step 3: Auditing {total_sections} sections", flush=True)
+            print(f"{'='*60}", flush=True)
+
+            section_idx = 0
+            for pass_def in passes:
+                pass_name = pass_def.get("name", "unknown")
+                sections = pass_def.get("asvs_sections", [])
+                include_files = pass_def.get("files", [])
+                domain_context = pass_def.get("domain_context", "")
+                pass_output_dir = f"{output_directory}/{pass_name}" if output_directory else pass_name
+                report_directories.append(pass_output_dir)
+
+                print(f"\n{'='*60}", flush=True)
+                print(f"Pass: {pass_name} ({len(sections)} sections)", flush=True)
+                print(f"{'='*60}", flush=True)
+
+                for section in sections:
+                    section_idx += 1
+                    print(f"\n[{section_idx}/{total_sections}] {section} ({pass_name})", flush=True)
+
+                    audit_output_text = None
+                    try:
+                        audit_result = await gofannon_client.call(
+                            agent_name="run_asvs_security_audit",
+                            input_dict={
+                                "inputText": json.dumps({
+                                    "namespaces": namespaces,
+                                    "asvs": section,
+                                    "includeFiles": include_files,
+                                    "domainContext": domain_context,
+                                    "severityThreshold": severity_threshold,
+                                    "falsePositiveGuidance": false_positive_guidance,
+                                })
+                            }
+                        )
+                        audit_output_text = audit_result.get("outputText", "")
+                        print(f"  Audit done: {len(audit_output_text)} chars", flush=True)
+                    except Exception as e:
+                        print(f"  Audit FAILED: {e}", flush=True)
+                        failures.append(f"{section} (audit): {e}")
+                        continue
+
+                    try:
+                        await gofannon_client.call(
+                            agent_name="add_markdown_file_to_github_directory",
+                            input_dict={
+                                "inputText": json.dumps({
+                                    "repo": output_repo,
+                                    "token": output_token,
+                                    "directory": pass_output_dir,
+                                    "filename": f"{section}.md",
+                                }),
+                                "commitMessage": f"ASVS {level or 'full'} audit: {section} ({pass_name})",
+                                "fileContents": audit_output_text,
+                            }
+                        )
+                        print(f"  Push OK", flush=True)
+                        successes.append(section)
+                    except Exception as e:
+                        print(f"  Push FAILED: {e}", flush=True)
+                        failures.append(f"{section} (push): {e}")
+                    all_outputs.append(audit_output_text)
+
+            # =============================================================
+            # Step 4: Consolidate
+            # =============================================================
+            if consolidate and successes:
+                print(f"\n{'='*60}", flush=True)
+                print(f"Step 4: Consolidating reports", flush=True)
+                print(f"{'='*60}", flush=True)
+                try:
+                    await gofannon_client.call(
+                        agent_name="consolidate_asvs_security_audit_reports",
+                        input_dict={
+                            "inputText": "\n".join([
+                                f"repo: {output_repo}",
+                                f"pat: {output_token}",
+                                f"directories: {', '.join(report_directories)}",
+                                f"output: {output_directory}",
+                            ]),
+                            "domainGroups": json.dumps(domain_groups),
+                            "level": level or "L3",
+                        }
+                    )
+                    print(f"  Consolidation done", flush=True)
+                except Exception as e:
+                    print(f"  Consolidation FAILED: {e}", flush=True)
+                    failures.append(f"consolidation: {e}")
+            elif not successes:
+                print(f"\n  Skipping consolidation — no successful audits", flush=True)
+
+        else:
+            # =============================================================
+            # NO-DISCOVER MODE: just audit all ASVS sections in the data
+            # store without domain scoping. Useful for small codebases.
+            # =============================================================
+            print(f"\n{'='*60}", flush=True)
+            print(f"Step 2: Loading ASVS sections (no discovery)", flush=True)
+            print(f"{'='*60}", flush=True)
+
+            # Get all ASVS sections from the data store, filtered by level
+            all_sections = []
+            try:
+                asvs_ns = data_store.use_namespace("asvs")
+                all_keys = asvs_ns.list_keys()
+                req_keys = [k for k in all_keys if k.startswith("asvs:requirements:")]
+                for rk in sorted(req_keys):
+                    section_id = rk.replace("asvs:requirements:", "")
+                    all_sections.append(section_id)
+            except Exception as e:
+                return {"outputText": f"Could not load ASVS sections: {e}"}
+
+            sections = filter_sections_by_level(all_sections)
+            print(f"  {len(sections)} sections to audit (level: {level or 'all'})", flush=True)
+
+            for i, section in enumerate(sections):
+                print(f"\n[{i+1}/{len(sections)}] Section {section}", flush=True)
+                audit_output_text = None
+                try:
+                    audit_result = await gofannon_client.call(
+                        agent_name="run_asvs_security_audit",
+                        input_dict={
+                            "inputText": json.dumps({
+                                "namespaces": namespaces,
+                                "asvs": section,
+                            })
+                        }
+                    )
+                    audit_output_text = audit_result.get("outputText", "")
+                except Exception as e:
+                    failures.append(f"{section} (audit): {e}")
+                    continue
+                try:
+                    await gofannon_client.call(
+                        agent_name="add_markdown_file_to_github_directory",
+                        input_dict={
+                            "inputText": json.dumps({
+                                "repo": output_repo,
+                                "token": output_token,
+                                "directory": output_directory,
+                                "filename": f"{section}.md",
+                            }),
+                            "commitMessage": f"ASVS audit: {section}",
+                            "fileContents": audit_output_text,
+                        }
+                    )
+                    successes.append(section)
+                except Exception as e:
+                    failures.append(f"{section} (push): {e}")
+                all_outputs.append(audit_output_text)
+
+        # =============================================================
+        # Summary
+        # =============================================================
+        print(f"\n{'='*60}", flush=True)
+        print(f"Complete: {len(successes)} succeeded, {len(failures)} failed", flush=True)
+        if failures:
+            for f in failures:
+                print(f"  - {f}", flush=True)
+        return {"outputText": "\n\n---\n\n".join(all_outputs)}
+    finally:
+        await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/run_asvs_security_audit.py b/repos/apache/ASVS/agents/run_asvs_security_audit.py
new file mode 100644
index 0000000..2e35855
--- /dev/null
+++ b/repos/apache/ASVS/agents/run_asvs_security_audit.py
@@ -0,0 +1,947 @@
+# run_asvs_security_audit
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+    mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+    http_client = httpx.AsyncClient()
+    try:
+        import json
+        import re
+        import fnmatch
+        from datetime import date
+        audit_date = date.today().strftime("%b %d, %Y")
+
+        # Parse input
+        input_text = input_dict.get("inputText", "")
+        params = {}
+        if input_text.strip().startswith('{'):
+            try:
+                params = json.loads(input_text)
+            except json.JSONDecodeError:
+                pass
+
+        if not params.get('namespace') and not params.get('namespaces'):
+            match = re.search(r'(?:namespaces?|ns)[:\s]+([^\n]+?)(?:\s+asvs|$)', input_text, re.IGNORECASE)
+            if match:
+                raw = match.group(1).strip()
+                if ',' in raw:
+                    params['namespaces'] = [n.strip() for n in raw.split(',')]
+                else:
+                    params['namespace'] = raw
+            else:
+                match = re.search(r'(?:namespace|ns)[:\s]+([^\s,]+)', input_text, re.IGNORECASE)
+                if match:
+                    params['namespace'] = match.group(1)
+
+        if not params.get('asvs') and not params.get('asvs_section'):
+            match = re.search(r'(?:asvs[-_]?section|asvs)[:\s]+(\d+(?:\.\d+)*)', input_text, re.IGNORECASE)
+            if match:
+                params['asvs'] = match.group(1)
+
+        namespaces = params.get('namespaces') or ([params.get('namespace')] if params.get('namespace') else [])
+        asvs = params.get('asvs') or params.get('asvs_section', '')
+
+        # --- NEW INPUTS ---
+        include_files = params.get('includeFiles', [])
+        severity_threshold = params.get('severityThreshold', '')
+        domain_context = params.get('domainContext', '')
+        false_positive_guidance = params.get('falsePositiveGuidance', [])
+
+        if not namespaces:
+            all_ns = data_store.list_namespaces()
+            file_ns = [ns for ns in all_ns if ns.startswith("files:")]
+            if file_ns:
+                namespaces = file_ns
+
+        if not namespaces:
+            return {"outputText": f"Error: No namespaces provided or found in data store. Available namespaces: {data_store.list_namespaces()}"}
+
+        if not asvs:
+            return {"outputText": "Error: No ASVS requirement specified. Please provide an asvs parameter (e.g., asvs: 5.3.2)"}
+
+        repo_name = "unknown"
+        for ns in namespaces:
+            if ns.startswith("files:"):
+                repo_name = ns.replace("files:", "")
+                break
+
+        print(f"Namespaces: {namespaces}", flush=True)
+        print(f"ASVS: {asvs}", flush=True)
+        if include_files:
+            print(f"File scope: {len(include_files)} patterns", flush=True)
+        if severity_threshold:
+            print(f"Severity threshold: {severity_threshold}", flush=True)
+
+        # =============================================================
+        # Model configuration
+        # =============================================================
+        SONNET_PROVIDER = "bedrock"
+        SONNET_MODEL = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+        SONNET_PARAMS = {"temperature": 0.7, "max_tokens": 16384}
+
+        OPUS_PROVIDER = "bedrock"
+        OPUS_MODEL = "us.anthropic.claude-opus-4-6-v1"
+        OPUS_PARAMS = {"temperature": 1, "reasoning_effort": "medium", "max_tokens": 64000}
+
+        SONNET_CONTEXT = get_context_window(SONNET_PROVIDER, SONNET_MODEL)
+        OPUS_CONTEXT = get_context_window(OPUS_PROVIDER, OPUS_MODEL)
+
+        sonnet_semaphore = asyncio.Semaphore(5)
+        opus_semaphore = asyncio.Semaphore(2)
+
+        cache_key_prefix = f"asvs-{asvs}-{'-'.join(namespaces)}"
+        relevance_cache_ns = data_store.use_namespace(f"audit-cache:relevance:{cache_key_prefix}")
+        inventory_cache_ns = data_store.use_namespace(f"audit-cache:inventory:{cache_key_prefix}")
+        analysis_cache_ns = data_store.use_namespace(f"audit-cache:analysis:{cache_key_prefix}")
+
+        # =============================================================
+        # Step 0: Load ASVS requirement context from data store
+        # =============================================================
+        print("\n=== Step 0: Loading ASVS requirement context ===", flush=True)
+        asvs_description = ""
+        try:
+            asvs_ns = data_store.use_namespace("asvs")
+            req = asvs_ns.get(f"asvs:requirements:{asvs}")
+            if req:
+                parts = [f"ASVS Requirement {asvs}"]
+                if req.get("req_description"):
+                    parts.append(f"Description: {req['req_description']}")
+                if req.get("level"):
+                    parts.append(f"Level: {req['level']}")
+                sec_id = req.get("section_id", "")
+                if sec_id:
+                    sec = asvs_ns.get(f"asvs:sections:{sec_id}")
+                    if sec:
+                        parts.append(f"Section: {sec.get('section_name', '')}")
+                        if sec.get("description"):
+                            parts.append(f"Section Description: {sec['description']}")
+                ch_id = req.get("chapter_id", "")
+                if ch_id:
+                    ch = asvs_ns.get(f"asvs:chapters:{ch_id}")
+                    if ch:
+                        parts.append(f"Chapter: {ch.get('chapter_name', '')}")
+                        if ch.get("control_objective"):
+                            parts.append(f"Control Objective: {ch['control_objective']}")
+                asvs_description = "\n".join(parts)
+                print(f"  Loaded: {asvs_description[:200]}", flush=True)
+            else:
+                print(f"  WARNING: No data found for asvs:requirements:{asvs}", flush=True)
+        except Exception as e:
+            print(f"  ERROR reading ASVS data store: {e}", flush=True)
+
+        if not asvs_description:
+            asvs_description = f"ASVS Requirement {asvs} (requirement description not available in data store)"
+
+        # =============================================================
+        # Step 1: Read files from data store (with optional pre-scoping)
+        # =============================================================
+        print("\n=== Step 1: Reading files from data store ===", flush=True)
+
+        all_files = {}
+        for ns in namespaces:
+            ns_store = data_store.use_namespace(ns)
+            keys = ns_store.list_keys()
+
+            # --- NEW: Pre-scope to includeFiles if provided ---
+            if include_files:
+                keys = [k for k in keys if any(
+                    fnmatch.fnmatch(k, pattern) for pattern in include_files
+                )]
+                print(f"Namespace '{ns}': {len(keys)} keys (scoped by includeFiles)", flush=True)
+            else:
+                print(f"Namespace '{ns}': {len(keys)} keys", flush=True)
+
+            file_contents = ns_store.get_many(keys) if keys else {}
+            for k, v in file_contents.items():
+                if v is not None:
+                    content = v if isinstance(v, str) else json.dumps(v, default=str) if v else ""
+                    all_files[k] = content
+
+        print(f"Total files loaded: {len(all_files)}", flush=True)
+
+        if not all_files:
+            return {"outputText": f"Error: No files found in namespaces {namespaces}. Available namespaces: {data_store.list_namespaces()}"}
+
+        # Skip non-code files
+        SKIP_DIRS = {
+            'node_modules', 'vendor', 'third_party', 'third-party',
+            'dist', 'build', 'out', 'target',
+            '__pycache__', '.pytest_cache', '.mypy_cache', 'coverage', '.next', '.nuxt',
+            'assets', 'images', 'img', 'static/images', 'static/fonts', 'static/webfonts',
+            'public/images', 'fonts', 'webfonts',
+            '.github/workflows',
+            'venv', '.venv', 'env', '.env',
+            '.git', '.idea', '.vscode',
+        }
+        SKIP_FILES = {
+            'package-lock.json', 'yarn.lock', 'poetry.lock', 'Cargo.lock',
+            'composer.lock', 'pnpm-lock.yaml', 'Gemfile.lock', 'uv.lock',
+            'LICENSE', 'LICENSE.md', 'LICENSE.txt',
+            'README.md', 'README.rst', 'README.txt', 'README',
+            'CHANGELOG.md', 'CHANGELOG', 'CONTRIBUTING.md', 'CODE_OF_CONDUCT.md',
+            '.gitignore', '.dockerignore', '.prettierrc', '.eslintrc', '.editorconfig',
+            '.npmrc', '.yarnrc',
+        }
+        SKIP_EXTENSIONS = {
+            '.min.js', '.min.css', '.bundle.js', '.bundle.css',
+            '.map', '.css', '.scss', '.less', '.sass', '.styl',
+            '.woff', '.woff2', '.ttf', '.eot', '.otf',
+            '.png', '.jpg', '.jpeg', '.gif', '.svg', '.ico', '.webp', '.bmp',
+            '.mp3', '.mp4', '.wav', '.avi', '.mov', '.webm', '.ogg',
+            '.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx',
+            '.zip', '.tar', '.gz', '.rar', '.7z', '.lock',
+            '.exe', '.dll', '.so', '.dylib', '.pyc', '.pyo', '.class', '.o', '.obj',
+        }
+
+        def should_skip_file(filepath):
+            path_lower = filepath.lower()
+            parts = filepath.split('/')
+            for part in parts[:-1]:
+                if part.lower() in SKIP_DIRS:
+                    return True
+            filename = parts[-1] if parts else ''
+            if filename in SKIP_FILES:
+                return True
+            for ext in SKIP_EXTENSIONS:
+                if path_lower.endswith(ext):
+                    return True
+            return False
+
+        filtered_files = {}
+        skipped_count = 0
+        for key, content in all_files.items():
+            if should_skip_file(key):
+                skipped_count += 1
+                continue
+            filtered_files[key] = content
+
+        print(f"Filtered: {len(filtered_files)} files to analyze, {skipped_count} skipped", flush=True)
+
+        if not filtered_files:
+            return {"outputText": f"Error: All {len(all_files)} files were filtered out. Check skip rules."}
+
+        # =============================================================
+        # Step 2: Relevance filtering (SKIPPED when includeFiles provided)
+        # =============================================================
+        if include_files:
+            print("\n=== Step 2: Relevance filtering SKIPPED (includeFiles provided) ===", flush=True)
+            relevant_files = filtered_files
+            relevance_scores = {path: 10 for path in filtered_files}
+            sorted_relevant = sorted(relevant_files.keys())
+        else:
+            print("\n=== Step 2: Relevance filtering (Sonnet, parallel) ===", flush=True)
+
+            cached_relevance = relevance_cache_ns.get("scores")
+            if cached_relevance:
+                relevance_scores = cached_relevance
+                print(f"  Using cached relevance scores for {len(relevance_scores)} files", flush=True)
+            else:
+                file_previews = {}
+                for path, content in filtered_files.items():
+                    lines = content.split('\n')
+                    file_previews[path] = '\n'.join(lines[:200])
+
+                SAFE_SONNET_LIMIT = int(SONNET_CONTEXT * 0.40)
+
+                relevance_prompt_template = f"""You are a security auditor performing file relevance filtering.
+
+ASVS Requirement: {asvs}
+{asvs_description}
+
+Below are file paths with previews (first ~200 lines) from a codebase.
+Rate each file's relevance to the ASVS requirement above on a scale of 0-10:
+- 10: Directly implements or should implement controls for this requirement
+- 7-9: Contains related security controls, data handling, or configuration
+- 4-6: May contain relevant patterns indirectly
+- 1-3: Unlikely to be relevant
+- 0: Definitely not relevant
+
+Return ONLY a JSON object mapping file paths to relevance scores (integer 0-10).
+Example: {{"src/auth.py": 9, "src/utils.py": 3}}
+
+FILES TO EVALUATE:
+"""
+
+                template_tokens = count_tokens(relevance_prompt_template, SONNET_PROVIDER, SONNET_MODEL)
+                preview_budget = SAFE_SONNET_LIMIT - template_tokens
+
+                preview_batches = []
+                current_batch = {}
+                current_tokens = 0
+                for path, preview in file_previews.items():
+                    entry = f"\n--- {path} ---\n{preview}\n"
+                    entry_tokens = count_tokens(entry, SONNET_PROVIDER, SONNET_MODEL)
+                    if current_tokens + entry_tokens > preview_budget and current_batch:
+                        preview_batches.append(current_batch)
+                        current_batch = {}
+                        current_tokens = 0
+                    current_batch[path] = entry
+                    current_tokens += entry_tokens
+                if current_batch:
+                    preview_batches.append(current_batch)
+
+                print(f"  Relevance filtering: {len(preview_batches)} batches", flush=True)
+
+                relevance_scores = {}
+
+                async def filter_batch(i, batch):
+                    async with sonnet_semaphore:
+                        entries_text = "".join(batch.values())
+                        prompt = relevance_prompt_template + entries_text
+                        messages = [{"role": "user", "content": prompt}]
+                        for attempt in range(2):
+                            try:
+                                content_resp, _ = await call_llm(
+                                    provider=SONNET_PROVIDER, model=SONNET_MODEL,
+                                    messages=messages, parameters=SONNET_PARAMS,
+                                    timeout=120,
+                                )
+                                json_match = re.search(r'\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}', content_resp, re.DOTALL)
+                                if json_match:
+                                    scores = json.loads(json_match.group())
+                                    print(f"    Batch {i+1}: scored {len(scores)} files", flush=True)
+                                    return scores
+                            except Exception as e:
+                                if attempt == 0:
+                                    print(f"    Batch {i+1} attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+                                    await asyncio.sleep(5)
+                                else:
+                                    print(f"    Batch {i+1} FAILED: {e}", flush=True)
+                        print(f"    WARNING: Batch {i+1} defaulting {len(batch)} files to score=5", flush=True)
+                        return {path: 5 for path in batch}
+
+                batch_results = await asyncio.gather(*[
+                    filter_batch(i, batch)
+                    for i, batch in enumerate(preview_batches)
+                ])
+                for scores in batch_results:
+                    relevance_scores.update(scores)
+
+                relevance_cache_ns.set("scores", relevance_scores)
+
+            relevant_files = {}
+            for path, content in filtered_files.items():
+                score = relevance_scores.get(path, 5)
+                if isinstance(score, (int, float)) and score >= 4:
+                    relevant_files[path] = content
+
+            if len(relevant_files) < 3 and filtered_files:
+                for path, content in filtered_files.items():
+                    score = relevance_scores.get(path, 5)
+                    if isinstance(score, (int, float)) and score >= 2:
+                        relevant_files[path] = content
+
+            sorted_relevant = sorted(relevant_files.keys(), key=lambda p: relevance_scores.get(p, 0), reverse=True)
+
+        print(f"  Relevant files: {len(relevant_files)} (from {len(filtered_files)})", flush=True)
+
+        # =============================================================
+        # Step 3: Code inventory (Sonnet, parallel, checkpointed)
+        # =============================================================
+        print("\n=== Step 3: Code inventory (Sonnet, parallel) ===", flush=True)
+
+        cached_inventory = inventory_cache_ns.get("result")
+        if cached_inventory:
+            code_inventory = cached_inventory
+            print(f"  Using cached inventory ({len(code_inventory)} chars)", flush=True)
+        else:
+            inventory_prompt_template = f"""You are a security code analyst. Extract a structured code inventory from each file below.
+
+For each file, produce:
+1. **Imports** — list all imports (especially security-related: auth, crypto, validators, sanitizers, path handling)
+2. **Classes** — name, base classes, key methods with signatures
+3. **Functions** — name, parameters (with types if available), decorators, line numbers (approximate)
+4. **Security-relevant patterns** — validators, auth checks, path operations, file operations, crypto operations
+5. **Routes/endpoints** — any URL routing, decorators like @app.route, @router.get, etc.
+6. **Configuration** — security-relevant config values, environment variables
+
+Return the inventory as structured markdown. Be thorough but concise.
+
+FILES:
+"""
+
+            SAFE_SONNET_LIMIT_INV = int(SONNET_CONTEXT * 0.40)
+            inv_template_tokens = count_tokens(inventory_prompt_template, SONNET_PROVIDER, SONNET_MODEL)
+            inv_budget = SAFE_SONNET_LIMIT_INV - inv_template_tokens
+
+            inv_batches = []
+            current_batch = {}
+            current_tokens = 0
+            for path in sorted_relevant:
+                content = relevant_files[path]
+                entry = f"\n--- {path} ---\n{content}\n"
+                entry_tokens = count_tokens(entry, SONNET_PROVIDER, SONNET_MODEL)
+                if entry_tokens > inv_budget:
+                    if current_batch:
+                        inv_batches.append(current_batch)
+                        current_batch = {}
+                        current_tokens = 0
+                    inv_batches.append({path: entry})
+                    continue
+                if current_tokens + entry_tokens > inv_budget and current_batch:
+                    inv_batches.append(current_batch)
+                    current_batch = {}
+                    current_tokens = 0
+                current_batch[path] = entry
+                current_tokens += entry_tokens
+            if current_batch:
+                inv_batches.append(current_batch)
+
+            print(f"  Code inventory: {len(inv_batches)} batches", flush=True)
+
+            async def inventory_batch(i, batch):
+                async with sonnet_semaphore:
+                    entries_text = "".join(batch.values())
+                    prompt = inventory_prompt_template + entries_text
+                    messages = [{"role": "user", "content": prompt}]
+
+                    msg_tokens = count_message_tokens(messages, SONNET_PROVIDER, SONNET_MODEL)
+                    if msg_tokens > int(SONNET_CONTEXT * 0.80):
+                        items = list(batch.items())
+                        mid = len(items) // 2
+                        results = []
+                        for sub_items in [items[:mid], items[mid:]]:
+                            sub_text = "".join([v for _, v in sub_items])
+                            sub_messages = [{"role": "user", "content": inventory_prompt_template + sub_text}]
+                            try:
+                                resp, _ = await call_llm(
+                                    provider=SONNET_PROVIDER, model=SONNET_MODEL,
+                                    messages=sub_messages, parameters=SONNET_PARAMS,
+                                    timeout=300,
+                                )
+                                results.append(resp)
+                            except Exception as e:
+                                print(f"    Inventory sub-batch failed: {e}", flush=True)
+                        return "\n\n".join(results)
+
+                    for attempt in range(2):
+                        try:
+                            resp, _ = await call_llm(
+                                provider=SONNET_PROVIDER, model=SONNET_MODEL,
+                                messages=messages, parameters=SONNET_PARAMS,
+                                timeout=300,
+                            )
+                            print(f"    Inventory batch {i+1} complete", flush=True)
+                            return resp
+                        except Exception as e:
+                            if attempt == 0:
+                                print(f"    Inventory batch {i+1} attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+                                await asyncio.sleep(5)
+                            else:
+                                print(f"    Inventory batch {i+1} FAILED: {e}", flush=True)
+                                return ""
+
+            inventory_results = await asyncio.gather(*[
+                inventory_batch(i, batch)
+                for i, batch in enumerate(inv_batches)
+            ])
+            code_inventory = "\n\n---\n\n".join([r for r in inventory_results if r])
+            inventory_cache_ns.set("result", code_inventory)
+
+        print(f"  Inventory total: {len(code_inventory)} chars", flush=True)
+
+        # =============================================================
+        # Step 4: Deep security analysis (Opus, checkpointed, with retry)
+        # =============================================================
+        print("\n=== Step 4: Deep security analysis (Opus) ===", flush=True)
+
+        # --- BUILD SYSTEM PROMPT WITH NEW INPUTS ---
+        analysis_system_prompt = f"""You are an expert application security auditor performing a comprehensive security audit against ASVS requirements.
+
+## ASVS Requirement Under Audit
+{asvs_description}
+"""
+
+        # Inject domain context if provided
+        if domain_context:
+            analysis_system_prompt += f"""
+## Domain Context
+{domain_context}
+"""
+
+        # Inject severity threshold if provided
+        if severity_threshold:
+            severity_levels = {"CRITICAL": 4, "HIGH": 3, "MEDIUM": 2, "LOW": 1}
+            threshold_val = severity_levels.get(severity_threshold.upper(), 0)
+            if threshold_val > 0:
+                included = [k for k, v in severity_levels.items() if v >= threshold_val]
+                analysis_system_prompt += f"""
+## Severity Threshold
+Only report findings at these severity levels: {', '.join(included)}.
+Do not include findings below {severity_threshold.upper()} severity.
+"""
+
+        # Inject false positive guidance if provided
+        if false_positive_guidance:
+            guidance_text = "\n".join(f"- {g}" for g in false_positive_guidance)
+            analysis_system_prompt += f"""
+## Known False Positive Patterns (DO NOT FLAG)
+The following patterns are intentional design decisions in this codebase. Do not report them as vulnerabilities:
+{guidance_text}
+"""
+
+        analysis_system_prompt += """
+## Audit Instructions
+
+Follow ALL of these analysis requirements:
+
+### Core Principle: Existence ≠ Application
+For each security control found:
+- Document where it's DEFINED
+- Map ALL entry points that should use it
+- Verify it's actually CALLED at each entry point
+- Flag coverage gaps (control exists but not applied = CRITICAL)
+
+### Gap Type Classification
+| Gap Type | Description | Severity |
+|----------|-------------|----------|
+| Type A | Entry point with NO control | Standard vulnerability |
+| Type B | Control EXISTS but NOT CALLED | CRITICAL (false confidence) |
+| Type C | Control CALLED but RESULT IGNORED | CRITICAL |
+| Type D | Control CALLED but AFTER sensitive operation | CRITICAL |
+
+### Related Function Analysis
+When you find a vulnerability, IMMEDIATELY search for:
+- Singular/plural variants
+- Sync/async variants
+- Public/private variants
+- Same-file functions with similar parameters or operations
+
+### False Positive Prevention
+Before finalizing each finding:
+1. Where does this input ACTUALLY originate? Is it truly user-controllable?
+2. Is there validation applied EARLIER in the call chain?
+3. Can an external attacker actually control this value?
+4. If you listed something as a positive pattern, don't also list it as a vulnerability
+
+### Exclusions
+Do NOT report:
+- Database-sourced values without injection path
+- Already-validated inputs
+- Developer tooling/test code
+- Issues requiring prior compromise
+- Theoretical issues without specific exploit
+- Test/example code
+
+### Output Requirements
+For each finding, provide:
+- Severity level (CRITICAL/HIGH/MEDIUM/LOW)
+- Finding ID (format: ASVS-{asvs.replace('.', '')}-SEV-NNN)
+- Exact file location and function name with line numbers
+- Vulnerable code quote
+- Data flow (source → sink → missing control)
+- Proof of concept (specific malicious request)
+- Impact description
+- Remediation with code example
+
+Also provide:
+- Security Controls Inventory with coverage analysis
+- Critical File Review tables
+- Positive Security Patterns
+- Architecture Observations
+- Prioritized Recommendations
+
+Be thorough but precise. If something is done correctly, acknowledge it as a positive pattern - don't invent issues."""
+
+        SAFE_OPUS_LIMIT = int(OPUS_CONTEXT * 0.40)
+        system_tokens = count_tokens(analysis_system_prompt, OPUS_PROVIDER, OPUS_MODEL)
+        inventory_section = f"\n\n## Code Inventory (extracted by pre-analysis)\n\n{code_inventory}\n\n"
+        inventory_tokens = count_tokens(inventory_section, OPUS_PROVIDER, OPUS_MODEL)
+        user_template = "## Source Code Files\n\nAnalyze the following files for security issues related to the ASVS requirement:\n\n"
+        user_template_tokens = count_tokens(user_template, OPUS_PROVIDER, OPUS_MODEL)
+
+        max_inv_tokens = int(SAFE_OPUS_LIMIT * 0.15)
+        if inventory_tokens > max_inv_tokens:
+            print(f"  Truncating inventory from {inventory_tokens} to {max_inv_tokens} tokens", flush=True)
+            inv_lines = code_inventory.split('\n')
+            truncated_inv = ""
+            for line in inv_lines:
+                candidate = truncated_inv + line + "\n"
+                if count_tokens(candidate, OPUS_PROVIDER, OPUS_MODEL) > max_inv_tokens:
+                    break
+                truncated_inv = candidate
+            inventory_section = f"\n\n## Code Inventory (extracted by pre-analysis, truncated)\n\n{truncated_inv}\n\n"
+            inventory_tokens = count_tokens(inventory_section, OPUS_PROVIDER, OPUS_MODEL)
+
+        opus_content_budget = SAFE_OPUS_LIMIT - system_tokens - inventory_tokens - user_template_tokens
+
+        print(f"  Opus content budget: {opus_content_budget} tokens", flush=True)
+
+        opus_batches = []
+        current_batch = {}
+        current_tokens = 0
+        for path in sorted_relevant:
+            content = relevant_files[path]
+            entry = f"\n### File: `{path}`\n```\n{content}\n```\n"
+            entry_tokens = count_tokens(entry, OPUS_PROVIDER, OPUS_MODEL)
+            if entry_tokens > opus_content_budget:
+                if current_batch:
+                    opus_batches.append(current_batch)
+                    current_batch = {}
+                    current_tokens = 0
+                opus_batches.append({path: entry})
+                continue
+            if current_tokens + entry_tokens > opus_content_budget and current_batch:
+                opus_batches.append(current_batch)
+                current_batch = {}
+                current_tokens = 0
+            current_batch[path] = entry
+            current_tokens += entry_tokens
+        if current_batch:
+            opus_batches.append(current_batch)
+
+        print(f"  Deep analysis: {len(opus_batches)} batches", flush=True)
+
+        async def analyze_batch(i, batch):
+            cache_key = f"batch-{i}"
+            cached = analysis_cache_ns.get(cache_key)
+            if cached:
+                print(f"    Opus batch {i+1}: cached", flush=True)
+                return cached
+
+            async with opus_semaphore:
+                entries_text = "".join(batch.values())
+                user_content = user_template + entries_text + inventory_section
+                messages = [
+                    {"role": "user", "content": analysis_system_prompt + "\n\n" + user_content}
+                ]
+
+                msg_tokens = count_message_tokens(messages, OPUS_PROVIDER, OPUS_MODEL)
+                limit = int(OPUS_CONTEXT * 0.80)
+                print(f"    Opus batch {i+1}/{len(opus_batches)}: {msg_tokens} tokens, {len(batch)} files", flush=True)
+
+                if msg_tokens > limit:
+                    items = list(batch.items())
+                    if len(items) > 1:
+                        mid = len(items) // 2
+                        results = []
+                        for half_label, half_items in [("a", items[:mid]), ("b", items[mid:])]:
+                            half_text = "".join([v for _, v in half_items])
+                            half_user = user_template + half_text + inventory_section
+                            half_messages = [{"role": "user", "content": analysis_system_prompt + "\n\n" + half_user}]
+                            for attempt in range(3):
+                                try:
+                                    resp, _ = await call_llm(
+                                        provider=OPUS_PROVIDER, model=OPUS_MODEL,
+                                        messages=half_messages, parameters=OPUS_PARAMS,
+                                        timeout=1800,
+                                    )
+                                    results.append(resp)
+                                    print(f"      Sub-batch {half_label} complete", flush=True)
+                                    break
+                                except Exception as e:
+                                    if attempt < 2:
+                                        wait = 15 * (attempt + 1)
+                                        print(f"      Sub-batch {half_label} attempt {attempt+1} failed ({type(e).__name__}), retrying in {wait}s...", flush=True)
+                                        await asyncio.sleep(wait)
+                                    else:
+                                        print(f"      Sub-batch {half_label} FAILED after 3 attempts: {e}", flush=True)
+                                        results.append(f"[Analysis failed for sub-batch {i+1}{half_label}: {str(e)[:200]}]")
+                        combined = "\n\n---\n\n".join(results)
+                        analysis_cache_ns.set(cache_key, combined)
+                        return combined
+                    else:
+                        key, entry_val = items[0]
+                        slim_messages = [{"role": "user", "content": analysis_system_prompt + "\n\n" + user_template + entry_val}]
+                        for attempt in range(3):
+                            try:
+                                resp, _ = await call_llm(
+                                    provider=OPUS_PROVIDER, model=OPUS_MODEL,
+                                    messages=slim_messages, parameters=OPUS_PARAMS,
+                                    timeout=1800,
+                                )
+                                analysis_cache_ns.set(cache_key, resp)
+                                return resp
+                            except Exception as e:
+                                if attempt < 2:
+                                    wait = 15 * (attempt + 1)
+                                    print(f"      Single file attempt {attempt+1} failed ({type(e).__name__}), retrying in {wait}s...", flush=True)
+                                    await asyncio.sleep(wait)
+                                else:
+                                    print(f"      Single file FAILED after 3 attempts: {e}", flush=True)
+                                    return f"[Analysis failed for {key}: {str(e)[:200]}]"
+
+                for attempt in range(3):
+                    try:
+                        resp, _ = await call_llm(
+                            provider=OPUS_PROVIDER, model=OPUS_MODEL,
+                            messages=messages, parameters=OPUS_PARAMS,
+                            timeout=1800,
+                        )
+                        analysis_cache_ns.set(cache_key, resp)
+                        print(f"    Opus batch {i+1} complete", flush=True)
+                        return resp
+                    except Exception as e:
+                        if attempt < 2:
+                            wait = 15 * (attempt + 1)
+                            print(f"    Opus batch {i+1} attempt {attempt+1} failed ({type(e).__name__}), retrying in {wait}s...", flush=True)
+                            await asyncio.sleep(wait)
+                        else:
+                            print(f"    Opus batch {i+1} FAILED after 3 attempts: {e}", flush=True)
+                            return f"[Analysis failed for batch {i+1}: {str(e)[:200]}]"
+
+        analysis_results = await asyncio.gather(*[
+            analyze_batch(i, batch)
+            for i, batch in enumerate(opus_batches)
+        ])
+
+        analysis_results = [r for r in analysis_results if r and not r.startswith("[Analysis failed")]
+        print(f"  Analysis complete: {len(analysis_results)} results", flush=True)
+
+        if not analysis_results:
+            return {"outputText": "Error: All analysis batches failed. No results to report."}
+
+        # =============================================================
+        # Step 5: Consolidation if needed (Sonnet)
+        # =============================================================
+        if len(analysis_results) > 1:
+            print(f"\n=== Step 5: Consolidating {len(analysis_results)} results (Sonnet) ===", flush=True)
+
+            CONSOLIDATION_TEMPLATE = f"""You are consolidating multiple security audit batch results into a single unified analysis.
+
+ASVS Requirement: {asvs}
+{asvs_description}
+
+## Consolidation Rules:
+1. DEDUPLICATE - Merge findings describing the same vulnerability
+2. CHECK CONTRADICTIONS - If something appears as a positive pattern in ANY batch, remove from findings
+3. VERIFY DATA ORIGINS - Remove findings where source is database/config without user injection path
+4. CONSISTENT SEVERITY - Ensure similar issues have same severity
+5. REMOVE OUT-OF-SCOPE - Exclude test files, dev scripts
+6. VERIFY COMPLETENESS - For each vulnerability, confirm related functions were checked
+
+Consolidate these analysis results into a single comprehensive security audit report.
+Preserve all specific findings with their exact code references, but merge duplicates.
+Include a "Consolidation Notes" section documenting what was removed/merged and why.
+
+BATCH RESULTS TO CONSOLIDATE:
+"""
+
+            consolidation_params = {**SONNET_PARAMS, "max_tokens": 16384}
+            template_tokens_cons = count_tokens(CONSOLIDATION_TEMPLATE, SONNET_PROVIDER, SONNET_MODEL)
+            max_cons_content = int(SONNET_CONTEXT * 0.40) - template_tokens_cons
+
+            consolidation_round = 0
+            MAX_CONSOLIDATION_ROUNDS = 5
+            batch_results = analysis_results[:]
+
+            while len(batch_results) > 1:
+                consolidation_round += 1
+                prev_count = len(batch_results)
+                next_level = []
+                group = []
+                group_tokens = 0
+
+                for result in batch_results:
+                    result_tokens = count_tokens(result, SONNET_PROVIDER, SONNET_MODEL)
+                    if group and (group_tokens + result_tokens) > max_cons_content:
+                        for attempt in range(2):
+                            try:
+                                cons_prompt = CONSOLIDATION_TEMPLATE + "\n---\n".join(group)
+                                consolidated, _ = await call_llm(
+                                    provider=SONNET_PROVIDER, model=SONNET_MODEL,
+                                    messages=[{"role": "user", "content": cons_prompt}],
+                                    parameters=consolidation_params,
+                                    timeout=300,
+                                )
+                                next_level.append(consolidated)
+                                break
+                            except Exception as e:
+                                if attempt == 0:
+                                    print(f"    Consolidation attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+                                    await asyncio.sleep(5)
+                                else:
+                                    print(f"    Consolidation failed, keeping individual results", flush=True)
+                                    next_level.extend(group)
+                        group = []
+                        group_tokens = 0
+                    group.append(result)
+                    group_tokens += result_tokens
+
+                if group:
+                    if len(group) == 1 and not next_level:
+                        next_level.append(group[0])
+                    else:
+                        for attempt in range(2):
+                            try:
+                                cons_prompt = CONSOLIDATION_TEMPLATE + "\n---\n".join(group)
+                                consolidated, _ = await call_llm(
+                                    provider=SONNET_PROVIDER, model=SONNET_MODEL,
+                                    messages=[{"role": "user", "content": cons_prompt}],
+                                    parameters=consolidation_params,
+                                    timeout=300,
+                                )
+                                next_level.append(consolidated)
+                                break
+                            except Exception as e:
+                                if attempt == 0:
+                                    print(f"    Consolidation attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+                                    await asyncio.sleep(5)
+                                else:
+                                    print(f"    Consolidation failed, keeping individual results", flush=True)
+                                    next_level.extend(group)
+
+                print(f"    Round {consolidation_round}: {prev_count} -> {len(next_level)}", flush=True)
+                batch_results = next_level
+
+                if len(batch_results) >= prev_count:
+                    print(f"    No progress, stopping", flush=True)
+                    break
+                if consolidation_round >= MAX_CONSOLIDATION_ROUNDS:
+                    print(f"    Max rounds reached", flush=True)
+                    break
+
+            consolidated_analysis = "\n\n---\n\n".join(batch_results)
+        else:
+            consolidated_analysis = analysis_results[0]
+
+        # =============================================================
+        # Step 6: Format report (Sonnet)
+        # =============================================================
+        print("\n=== Step 6: Formatting report (Sonnet) ===", flush=True)
+
+        def count_findings(content):
+            counts = {"Critical": 0, "High": 0, "Medium": 0, "Low": 0}
+            counts["Critical"] = len(re.findall(r'###\s*\[CRITICAL\]', content, re.IGNORECASE))
+            counts["High"] = len(re.findall(r'###\s*\[HIGH\]', content, re.IGNORECASE))
+            counts["Medium"] = len(re.findall(r'###\s*\[MEDIUM\]', content, re.IGNORECASE))
+            counts["Low"] = len(re.findall(r'###\s*\[LOW\]', content, re.IGNORECASE))
+            if sum(counts.values()) == 0:
+                counts["Critical"] += len(re.findall(r'###[^\n]*-CRIT-', content, re.IGNORECASE))
+                counts["High"] += len(re.findall(r'###[^\n]*-HIGH-', content, re.IGNORECASE))
+                counts["Medium"] += len(re.findall(r'###[^\n]*-MED-', content, re.IGNORECASE))
+                counts["Low"] += len(re.findall(r'###[^\n]*-LOW-', content, re.IGNORECASE))
+            if sum(counts.values()) == 0:
+                counts["Critical"] += len(re.findall(r'\*\*Severity\*\*:\s*Critical', content, re.IGNORECASE))
+                counts["High"] += len(re.findall(r'\*\*Severity\*\*:\s*High', content, re.IGNORECASE))
+                counts["Medium"] += len(re.findall(r'\*\*Severity\*\*:\s*Medium', content, re.IGNORECASE))
+                counts["Low"] += len(re.findall(r'\*\*Severity\*\*:\s*Low', content, re.IGNORECASE))
+            return counts
+
+        findings_count = count_findings(consolidated_analysis)
+        file_list = "\n".join([f"- `{p}` (relevance: {relevance_scores.get(p, '?')})" for p in sorted_relevant])
+
+        format_prompt = f"""You are formatting a security audit report into clean, professional markdown.
+
+ASVS Requirement: {asvs}
+{asvs_description}
+
+## Formatting Requirements
+
+Format the analysis below into the following structure. Preserve ALL specific findings, code references,
+line numbers, and technical details exactly as provided. Do not add, remove, or modify any findings.
+
+Required structure:
+1. **Executive Summary** — with these EXACT metadata values (do NOT invent names):
+   - **Repository:** {repo_name}
+   - **Audit Date:** {audit_date}
+   - **Auditor:** Tooling Agents
+   - **ASVS Requirement:** {asvs} — {asvs_description[:500]}
+   - Files analyzed: {len(relevant_files)} relevant out of {len(filtered_files)} total, {skipped_count} skipped
+   - Finding counts table
+2. **Security Controls Inventory** — each control with location, purpose, coverage status
+3. **Critical File Review** — tables for key files showing all security-sensitive functions reviewed
+4. **Findings** — grouped by severity (Critical → High → Medium → Low), each with:
+   - ID (format: ASVS-{asvs.replace('.', '')}-SEV-NNN)
+   - Location with file, function, line numbers
+   - Related functions checked
+   - Description, vulnerable code, data flow, PoC, impact, remediation
+5. **Positive Security Patterns** — what's done well
+6. **Architecture Observations**
+7. **Recommendations Summary** — Immediate/Short-term/Long-term
+8. **Appendix: Files Analyzed** — collapsible list
+
+Finding counts detected: Critical={findings_count['Critical']}, High={findings_count['High']}, Medium={findings_count['Medium']}, Low={findings_count['Low']}
+
+## Files Analyzed
+{file_list}
+
+## Analysis to Format
+
+{consolidated_analysis}"""
+
+        format_messages = [{"role": "user", "content": format_prompt}]
+        format_tokens = count_message_tokens(format_messages, SONNET_PROVIDER, SONNET_MODEL)
+        sonnet_limit = int(SONNET_CONTEXT * 0.80)
+
+        print(f"  Format prompt: {format_tokens} tokens (limit: {sonnet_limit})", flush=True)
+
+        if format_tokens <= sonnet_limit:
+            for attempt in range(2):
+                try:
+                    final_report, _ = await call_llm(
+                        provider=SONNET_PROVIDER, model=SONNET_MODEL,
+                        messages=format_messages, parameters=SONNET_PARAMS,
+                        timeout=600,
+                    )
+                    break
+                except Exception as e:
+                    if attempt == 0:
+                        print(f"  Formatting attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+                        await asyncio.sleep(5)
+                    else:
+                        print(f"  Formatting failed, using raw analysis: {e}", flush=True)
+                        final_report = consolidated_analysis
+        else:
+            print(f"  Format prompt too large, using raw analysis with header", flush=True)
+            header = f"""# Security Audit Report
+
+## Executive Summary
+
+| Field | Value |
+|-------|-------|
+| **Repository** | {repo_name} |
+| **Audit Date** | {audit_date} |
+| **Auditor** | Tooling Agents |
+| **ASVS Requirement** | {asvs} — {asvs_description[:500]} |
+| **Files Analyzed** | {len(relevant_files)} relevant / {len(filtered_files)} total |
+| **Files Skipped** | {skipped_count} |
+| **Analysis Batches** | {len(opus_batches)} |
+
+### Findings Overview
+
+| Severity | Count |
+|----------|-------|
+| 🔴 Critical | {findings_count['Critical']} |
+| 🟠 High | {findings_count['High']} |
+| 🟡 Medium | {findings_count['Medium']} |
+| 🟢 Low | {findings_count['Low']} |
+
+---
+
+"""
+            appendix = f"""
+
+---
+
+## Appendix: Files Analyzed
+
+<details>
+<summary>Click to expand ({len(relevant_files)} files)</summary>
+
+{file_list}
+
+</details>
+"""
+            final_report = header + consolidated_analysis + appendix
+
+        # Save report
+        report_ns = data_store.use_namespace("audit-reports")
+        report_key = f"asvs-{asvs}-{'-'.join(namespaces)}"
+        report_ns.set(report_key, {
+            "asvs": asvs,
+            "namespaces": namespaces,
+            "files_analyzed": len(relevant_files),
+            "files_total": len(filtered_files),
+            "files_skipped": skipped_count,
+            "findings": findings_count,
+            "report": final_report[:50000],
+        })
+
+        print(f"\n=== Done: {len(final_report)} chars ===", flush=True)
+        return {"outputText": final_report}
+
+    finally:
+        await http_client.aclose()
\ No newline at end of file


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org