You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/04/21 06:59:59 UTC
(tooling-agents) branch main updated: Remove ATR-isms, add download, discover, and consolidate to pipeline
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
The following commit(s) were added to refs/heads/main by this push:
new d57e15e Remove ATR-isms, add download, discover, and consolidate to pipeline
d57e15e is described below
commit d57e15e8edaae556e2961ed0ce83e4c2e6bee7d7
Author: Andrew Musselman <an...@gmail.com>
AuthorDate: Mon Apr 20 23:59:50 2026 -0700
Remove ATR-isms, add download, discover, and consolidate to pipeline
---
repos/apache/ASVS/README.md | 199 +++++
.../add_markdown_file_to_github_directory.py | 170 ++++
.../consolidate_asvs_security_audit_reports.py | 953 +++++++++++++++++++++
.../ASVS/agents/discover_codebase_architecture.py | 366 ++++++++
.../agents/download_github_repo_to_data_store.py | 149 ++++
.../agents/orchestrate_asvs_audit_to_github.py | 368 ++++++++
.../apache/ASVS/agents/run_asvs_security_audit.py | 947 ++++++++++++++++++++
7 files changed, 3152 insertions(+)
diff --git a/repos/apache/ASVS/README.md b/repos/apache/ASVS/README.md
new file mode 100644
index 0000000..e463c22
--- /dev/null
+++ b/repos/apache/ASVS/README.md
@@ -0,0 +1,199 @@
+# ASVS Security Audit Pipeline
+
+## Pipeline Flow
+
+```
+orchestrate_asvs_audit_to_github (single entry point)
+ │
+ ├──▶ download_github_repo_to_datastore (once — fetches source code)
+ │
+ ├──▶ discover_codebase_architecture (once — generates audit plan)
+ │
+ ├──▶ run_asvs_security_audit (× N sections)
+ │ └──▶ add_markdown_file_to_github_directory (× N)
+ │
+ └──▶ consolidate_asvs_security_audit_reports (once — final report)
+```
+
+Pre-requisites (one-time, outside the pipeline):
+- ASVS requirements loaded into the `asvs` data store namespace
+- Optionally: audit guidance loaded via `fetch_audit_guidance`
+
+## Quick Start
+
+Run `orchestrate_asvs_audit_to_github` with:
+
+| Input | Value |
+|---|---|
+| `sourceRepo` | `apache/airflow` |
+| `outputRepo` | `apache/tooling-runbooks` |
+| `outputToken` | `ghp_...` |
+| `outputDirectory` | `ASVS/reports` |
+| `level` | `L2` |
+| `severityThreshold` | `HIGH` |
+
+That's it. The orchestrator downloads the code, fetches the latest commit
+hash, discovers the architecture, runs all audits, pushes reports, and
+consolidates. The output directory is automatically extended with the
+repo name and commit hash: `ASVS/reports/airflow/da901ba`.
+
+To audit a subdirectory of a large repo:
+
+| Input | Value |
+|---|---|
+| `sourceRepo` | `apache/airflow/airflow-core/src` |
+
+This downloads only files under `airflow-core/src/`.
+
+## Reports Location
+
+```
+ASVS/reports/airflow/da901ba/ ← auto-generated from repo + HEAD commit
+├── auth_identity/ ← per-domain subdirectories
+│ ├── 6.1.1.md
+│ └── 7.1.1.md
+├── secrets_crypto/
+│ └── 9.1.1.md
+├── consolidated.md ← THE REPORT
+└── issues.md ← GitHub issues, one per finding
+```
+
+## Level System
+
+| Input | Requirements audited |
+|---|---|
+| `L1` | L1 only |
+| `L2` | L1 + L2 |
+| `L3` | L1 + L2 + L3 (all) |
+
+## Severity Threshold
+
+| Input | Findings included |
+|---|---|
+| `CRITICAL` | Critical only |
+| `HIGH` | Critical + High |
+| `MEDIUM` | Critical + High + Medium |
+| (empty) | All findings |
+
+---
+
+## Agent Reference
+
+All gofannon inputs are strings.
+
+### 1. orchestrate_asvs_audit_to_github
+
+The main entry point. Calls all other agents.
+
+| Input | Required | Description |
+|---|---|---|
+| `sourceRepo` | yes | Source code to audit. Accepts `owner/repo`, `owner/repo/subdir`, or full GitHub URL like `https://github.com/owner/repo/tree/branch/subdir` |
+| `sourceToken` | no | PAT for private source repos |
+| `supplementalData` | no | Extra data store namespaces, comma-separated (e.g., `audit_guidance`) |
+| `outputRepo` | yes | GitHub repo for reports (`owner/repo`) |
+| `outputToken` | yes | PAT with write access to output repo |
+| `outputDirectory` | yes | Base directory — repo name and commit hash are appended automatically |
+| `discover` | no | `"true"` or `"false"` (default `"true"`) |
+| `level` | no | `"L1"`, `"L2"`, or `"L3"` |
+| `severityThreshold` | no | `"CRITICAL"`, `"HIGH"`, `"MEDIUM"`, or empty |
+| `consolidate` | no | `"true"` or `"false"` (default `"true"`) |
+
+**Output:** `outputText` — combined audit results.
+
+**Namespace derivation:** The orchestrator parses `sourceRepo` to derive the code namespace automatically. `apache/airflow` → `files:apache/airflow`. If `supplementalData` is `audit_guidance`, the namespace list becomes `["files:apache/airflow", "audit_guidance"]`.
+
+**Agents called:**
+1. `download_github_repo_to_datastore` — downloads source code
+2. `discover_codebase_architecture` — generates audit plan (if `discover="true"`)
+3. `run_asvs_security_audit` — once per ASVS section
+4. `add_markdown_file_to_github_directory` — once per section
+5. `consolidate_asvs_security_audit_reports` — final report (if `consolidate="true"`)
+
+---
+
+### 2. download_github_repo_to_datastore
+
+Downloads a GitHub repo (or subdirectory) into the data store.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | `owner/repo` or `owner/repo/subdir`, optional token on next line |
+
+**Output:** `outputText` — download summary.
+
+Stores files in namespace `files:{owner}/{repo}`. When a path prefix is included (e.g., `apache/airflow/airflow-core/src`), only files under that path are downloaded. File paths in the data store are preserved as full repo-relative paths.
+
+---
+
+### 3. discover_codebase_architecture
+
+Scans codebase, generates domains + file lists + false positive guidance.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputNamespace` | yes | Data store namespace(s) with code, comma-separated |
+
+**Output:** `outputText` — JSON passConfig containing `passes`, `domain_groups`, `false_positive_guidance`.
+
+---
+
+### 4. run_asvs_security_audit
+
+Audits code against a single ASVS requirement.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | JSON string (see fields below) |
+
+JSON fields inside `inputText`:
+
+| Field | Required | Description |
+|---|---|---|
+| `namespaces` | yes | Array of data store namespaces |
+| `asvs` | yes | ASVS section (e.g., `"6.1.1"`) |
+| `includeFiles` | no | Array of file glob patterns — skips relevance filtering |
+| `domainContext` | no | Architecture context for Opus prompt |
+| `severityThreshold` | no | Minimum severity to report |
+| `falsePositiveGuidance` | no | Array of patterns to suppress |
+
+**Output:** `outputText` — markdown audit report.
+
+---
+
+### 5. consolidate_asvs_security_audit_reports
+
+Reads per-section reports from GitHub, deduplicates, produces consolidated report + issues.
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | Multi-line: `repo:`, `pat:`, `directories:`, `output:` |
+| `domainGroups` | no | JSON string mapping domain names to ASVS section arrays |
+| `level` | no | `"L1"`, `"L2"`, or `"L3"` (default `"L3"`) |
+
+**Output:** `outputText` — summary. Pushes `consolidated.md` and `issues.md` to GitHub.
+
+---
+
+### 6. add_markdown_file_to_github_directory
+
+| Input | Required | Description |
+|---|---|---|
+| `inputText` | yes | JSON with `repo`, `token`, `directory`, `filename` |
+| `commitMessage` | yes | Git commit message |
+| `fileContents` | yes | Markdown content |
+
+**Output:** `outputText` — GitHub API response.
+
+---
+
+## Troubleshooting
+
+**"sourceRepo is required"** — Provide `sourceRepo` as `owner/repo`, `owner/repo/subdir`, or a full GitHub URL.
+
+**"No ASVS sections match level L1"** — Check that ASVS requirement entries have a `level` field.
+
+**Download takes too long** — Use a path prefix to scope: `apache/airflow/airflow-core/src` instead of `apache/airflow`.
+
+**Too many findings** — Use `severityThreshold="HIGH"` or `"CRITICAL"`.
+
+**Reports not on GitHub** — Check `outputToken` has write access to `outputRepo`.
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/add_markdown_file_to_github_directory.py b/repos/apache/ASVS/agents/add_markdown_file_to_github_directory.py
new file mode 100644
index 0000000..4d29665
--- /dev/null
+++ b/repos/apache/ASVS/agents/add_markdown_file_to_github_directory.py
@@ -0,0 +1,170 @@
+# add_markdown_file_to_github_directory
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+import litellm
+import httpx
+
+async def run(input_dict, tools):
+ mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+ http_client = httpx.AsyncClient()
+ try:
+ import json
+ import re
+ import base64
+ from datetime import datetime, timezone
+
+ input_text = (input_dict.get("inputText") or "").strip()
+ commit_message = input_dict.get("commitMessage") or "Add markdown file"
+ file_contents = input_dict.get("fileContents") or ""
+
+ def _maybe_json(s: str):
+ try:
+ return json.loads(s)
+ except Exception:
+ return None
+
+ def _strip_quotes(s: str) -> str:
+ s = s.strip()
+ if (len(s) >= 2) and ((s[0] == s[-1]) and s[0] in ("'", '"')):
+ return s[1:-1].strip()
+ return s
+
+ def _find_kv(text: str, key: str):
+ patterns = [
+ rf'(?im)^\s*{re.escape(key)}\s*:\s*(.+?)\s*$',
+ rf'(?im)^\s*{re.escape(key)}\s*=\s*(.+?)\s*$',
+ rf'(?is)"{re.escape(key)}"\s*:\s*"([^"]+)"',
+ rf"(?is)'{re.escape(key)}'\s*:\s*'([^']+)'",
+ ]
+ for pat in patterns:
+ m = re.search(pat, text)
+ if m:
+ return _strip_quotes(m.group(1))
+ return None
+
+ def _parse_owner_repo(text: str):
+ repo = _find_kv(text, "repo") or _find_kv(text, "repository") or _find_kv(text, "full_name")
+ if repo and "/" in repo:
+ parts = repo.strip().strip("/").split("/")
+ if len(parts) >= 2:
+ return parts[-2], parts[-1]
+ m = re.search(r'(?i)\bgithub\.com/([^/\s]+)/([^/\s#?]+)', text)
+ if m:
+ return m.group(1), m.group(2)
+ m = re.search(r'(?i)\b([A-Za-z0-9_.-]+)/([A-Za-z0-9_.-]+)\b', text)
+ if m:
+ return m.group(1), m.group(2)
+ return None, None
+
+ cfg = _maybe_json(input_text)
+ if not isinstance(cfg, dict):
+ cfg = {}
+
+ token = (
+ cfg.get("token") or cfg.get("github_token") or cfg.get("githubToken")
+ or _find_kv(input_text, "token") or _find_kv(input_text, "github_token")
+ or _find_kv(input_text, "githubToken")
+ )
+
+ api_base = (
+ cfg.get("api_base") or cfg.get("apiBase") or cfg.get("github_api_base")
+ or _find_kv(input_text, "api_base") or _find_kv(input_text, "apiBase")
+ or _find_kv(input_text, "github_api_base") or "https://api.github.com"
+ ).rstrip("/")
+
+ owner = cfg.get("owner") or _find_kv(input_text, "owner")
+ repo_name = cfg.get("repo_name") or cfg.get("repo") or cfg.get("repository") or _find_kv(input_text, "repo_name")
+ if repo_name and isinstance(repo_name, str) and "/" in repo_name and not owner:
+ owner, repo_name = repo_name.split("/", 1)
+
+ if not owner or not repo_name:
+ o2, r2 = _parse_owner_repo(input_text)
+ owner = owner or o2
+ repo_name = repo_name or r2
+
+ directory = (
+ cfg.get("directory") or cfg.get("dir") or cfg.get("path")
+ or _find_kv(input_text, "directory") or _find_kv(input_text, "dir")
+ or _find_kv(input_text, "path") or ""
+ ).strip()
+
+ branch = cfg.get("branch") or _find_kv(input_text, "branch") or None
+
+ file_name = (
+ cfg.get("fileName") or cfg.get("filename") or cfg.get("file_name")
+ or _find_kv(input_text, "fileName") or _find_kv(input_text, "filename")
+ or _find_kv(input_text, "file_name")
+ )
+
+ file_path = (
+ cfg.get("filePath") or cfg.get("filepath")
+ or _find_kv(input_text, "filePath") or _find_kv(input_text, "filepath")
+ )
+
+ if not token:
+ return {"outputText": "Error: No GitHub token found in inputText (expected keys like token/github_token)."}
+ if not owner or not repo_name:
+ return {"outputText": "Error: Could not determine GitHub owner/repo from inputText (expected owner+repo or owner/repo)."}
+ if not directory and not file_path:
+ return {"outputText": "Error: No target directory/path provided in inputText (expected directory/dir/path or filePath)."}
+ if not file_contents:
+ return {"outputText": "Error: fileContents is empty."}
+
+ if file_path:
+ final_path = file_path.strip().lstrip("/")
+ else:
+ dir_clean = directory.strip().strip("/")
+ if file_name:
+ name = file_name.strip().lstrip("/")
+ else:
+ ts = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+ name = f"new_file_{ts}.md"
+ if not name.lower().endswith(".md"):
+ name = f"{name}.md"
+ final_path = f"{dir_clean}/{name}" if dir_clean else name
+
+ headers = {
+ "Authorization": f"token {token}",
+ "Accept": "application/vnd.github+json",
+ }
+
+ get_url = f"{api_base}/repos/{owner}/{repo_name}/contents/{final_path}"
+ params = {}
+ if branch:
+ params["ref"] = branch
+
+ existing_sha = None
+ existing_resp = await http_client.get(get_url, headers=headers, params=params)
+ if existing_resp.status_code == 200:
+ try:
+ existing_json = existing_resp.json()
+ existing_sha = existing_json.get("sha")
+ except Exception:
+ pass
+
+ put_url = f"{api_base}/repos/{owner}/{repo_name}/contents/{final_path}"
+ payload = {
+ "message": commit_message,
+ "content": base64.b64encode(file_contents.encode("utf-8")).decode("utf-8"),
+ }
+ if existing_sha:
+ payload["sha"] = existing_sha
+ if branch:
+ payload["branch"] = branch
+
+ resp = await http_client.put(put_url, headers=headers, json=payload)
+ try:
+ resp.raise_for_status()
+ except Exception:
+ try:
+ return {"outputText": json.dumps(resp.json(), indent=2, sort_keys=True)}
+ except Exception:
+ return {"outputText": resp.text}
+
+ try:
+ return {"outputText": json.dumps(resp.json(), indent=2, sort_keys=True)}
+ except Exception:
+ return {"outputText": resp.text}
+
+ finally:
+ await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/consolidate_asvs_security_audit_reports.py b/repos/apache/ASVS/agents/consolidate_asvs_security_audit_reports.py
new file mode 100644
index 0000000..c54c76d
--- /dev/null
+++ b/repos/apache/ASVS/agents/consolidate_asvs_security_audit_reports.py
@@ -0,0 +1,953 @@
+# consolidate_asvs_security_audit_reports
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+async def run(input_dict, tools):
+ mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+ http_client = httpx.AsyncClient()
+ try:
+ import json
+ import re
+ import base64
+ from datetime import date
+
+ audit_date = date.today().strftime("%b %d, %Y")
+
+ # =============================================================
+ # Markdown HTML sanitizer
+ # =============================================================
+ def sanitize_md_html(text):
+ if not text:
+ return text
+ parts = re.split(r'(```[\s\S]*?```)', text)
+ out = []
+ for part in parts:
+ if part.startswith('```'):
+ out.append(part)
+ continue
+ stash = []
+ def _stash(m):
+ stash.append(m.group(0))
+ return f'\x00IC{len(stash)-1}\x00'
+ s = re.sub(r'`[^`\n]+`', _stash, part)
+ s = re.sub(r'<(/?\w[^>]*)>', r'<\1>', s)
+ for j, code in enumerate(stash):
+ s = s.replace(f'\x00IC{j}\x00', code)
+ out.append(s)
+ return ''.join(out)
+
+ # =============================================================
+ # Parse inputs
+ # =============================================================
+ input_text = input_dict.get("inputText", "")
+ domain_groups_raw = input_dict.get("domainGroups", "")
+ level = input_dict.get("level", "L3")
+
+ # Normalize level
+ level = level.strip().upper()
+ if level and not level.startswith("L"):
+ level = f"L{level}"
+ LEVEL_ORDER = {"L1": 1, "L2": 2, "L3": 3}
+ max_level_num = LEVEL_ORDER.get(level, 3)
+
+ print(f"DEBUG raw input: {repr(input_text[:500])}", flush=True)
+ lines = input_text.strip().split("\n")
+ owner_repo = ""
+ pat = ""
+ directories_raw = ""
+ output_directory = ""
+ for line in lines:
+ line = line.strip()
+ if not line:
+ continue
+ if ":" in line:
+ key, _, value = line.partition(":")
+ key = key.strip().lower()
+ value = value.strip()
+ if key in ("owner/repo", "repo", "repository", "owner_repo"):
+ owner_repo = value
+ elif key in ("pat", "token", "github_token", "personal_access_token"):
+ pat = value
+ elif key in ("directories", "dirs", "paths", "directory", "dir", "path"):
+ directories_raw = value
+ elif key in ("output", "output_directory", "output_dir"):
+ output_directory = value
+
+ directories = [d.strip().strip("/") for d in directories_raw.split(",") if d.strip()]
+
+ if not owner_repo or not pat or not directories:
+ parts = input_text.strip().split()
+ tokens = [p.strip() for p in parts if p.strip()]
+ if not owner_repo:
+ for t in tokens:
+ if "/" in t and not t.startswith("/") and not t.startswith("ghp_") and "." not in t.split("/")[0]:
+ owner_repo = t
+ break
+ if not pat:
+ for t in tokens:
+ if t.startswith("ghp_") or t.startswith("github_pat_"):
+ pat = t
+ break
+
+ assert owner_repo, "Could not parse owner/repo from input"
+ assert pat, "Could not parse PAT from input"
+ assert directories, "Could not parse directories from input"
+
+ owner, repo = owner_repo.split("/", 1)
+
+ if not output_directory:
+ output_directory = "/".join(directories[0].split("/")[:-1])
+ output_directory = output_directory.strip("/")
+
+ # Output filenames — no level suffix
+ consolidated_filename = "consolidated.md"
+ issues_filename = "issues.md"
+
+ print(f"Repository: {owner}/{repo}")
+ print(f"Directories: {directories}")
+ print(f"Level: {level}")
+ print(f"Output: {output_directory}")
+
+ # =============================================================
+ # Model configuration
+ # =============================================================
+ FAST_PROVIDER = "bedrock"
+ FAST_MODEL = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+ FAST_PARAMS = {"temperature": 0.7, "max_tokens": 16384}
+
+ HEAVY_PROVIDER = "bedrock"
+ HEAVY_MODEL = "us.anthropic.claude-opus-4-6-v1"
+ HEAVY_PARAMS = {"temperature": 1, "reasoning_effort": "high", "max_tokens": 128000}
+
+ FAST_CONTEXT_WINDOW = get_context_window(FAST_PROVIDER, FAST_MODEL)
+ HEAVY_CONTEXT_WINDOW = get_context_window(HEAVY_PROVIDER, HEAVY_MODEL)
+
+ dirs_key = "+".join(sorted(directories))
+ extraction_ns = data_store.use_namespace(f"extraction:{owner}/{repo}/{dirs_key}")
+ consolidation_ns = data_store.use_namespace(f"consolidation:{owner}/{repo}/{dirs_key}")
+
+ GITHUB_API = "https://api.github.com"
+ headers = {
+ "Authorization": f"token {pat}",
+ "Accept": "application/vnd.github.v3+json",
+ "X-GitHub-Api-Version": "2022-11-28",
+ }
+ repo_resp = await http_client.get(f"{GITHUB_API}/repos/{owner}/{repo}", headers=headers)
+ repo_data = repo_resp.json()
+ default_branch = repo_data.get("default_branch", "main")
+
+ # ============================================================
+ # PHASE 1: Read All Reports from All Directories
+ # ============================================================
+ print("\n=== PHASE 1: Reading all reports ===")
+
+ reports = {}
+
+ for directory in directories:
+ print(f"\n Reading reports from {directory}...")
+ contents_resp = await http_client.get(
+ f"{GITHUB_API}/repos/{owner}/{repo}/contents/{directory}",
+ headers=headers,
+ params={"ref": default_branch},
+ )
+ if contents_resp.status_code != 200:
+ print(f" WARNING: Failed to list {directory}: {contents_resp.status_code}")
+ continue
+
+ dir_contents = contents_resp.json()
+ report_files = []
+ for item in dir_contents:
+ if item["type"] == "file" and item["name"].endswith(".md"):
+ if not item["name"].startswith("consolidated") and \
+ not item["name"].startswith("issues"):
+ report_files.append(item)
+
+ print(f" Found {len(report_files)} report files")
+
+ for item in report_files:
+ file_resp = await http_client.get(
+ f"{GITHUB_API}/repos/{owner}/{repo}/contents/{directory}/{item['name']}",
+ headers=headers,
+ params={"ref": default_branch},
+ )
+ if file_resp.status_code == 200:
+ file_data = file_resp.json()
+ content = base64.b64decode(file_data["content"]).decode("utf-8", errors="replace")
+ report_key = item['name']
+ reports[report_key] = content
+ print(f" Read {report_key} ({len(content)} chars)")
+
+ total_reports = len(reports)
+ print(f"\nSuccessfully read {total_reports} reports")
+
+ # ============================================================
+ # PHASE 2: Extract Findings (Sonnet, parallel, checkpointed)
+ # ============================================================
+ print("\n=== PHASE 2: Extracting findings ===")
+
+ EXTRACTION_PROMPT_TEMPLATE = """You are a security finding extractor. Given an ASVS audit report, extract ALL findings into structured JSON.
+
+For each finding, capture:
+- source_report: the report filename
+- finding_id: the ID used in the source report
+- severity: Critical/High/Medium/Low/Informational
+- title: short descriptive title
+- description: full description of the vulnerability
+- cwe: CWE identifier if mentioned
+- asvs_section: the ASVS section being audited (e.g., "8.2.2")
+- affected_files: list of objects with "file" and "line" keys
+- recommended_remediation: the recommended fix
+- positive_controls: list of any positive security controls noted
+
+Also extract:
+- asvs_status: overall status of this ASVS section (Pass/Fail/Partial/N/A)
+- asvs_section_title: the title/topic of this ASVS section if mentioned
+
+If the report has NO findings, return an empty findings list but still set asvs_status.
+
+Return ONLY valid JSON:
+{
+ "source_report": "filename.md",
+ "asvs_section": "X.Y.Z",
+ "asvs_section_title": "Title",
+ "asvs_status": "Pass|Fail|Partial|N/A",
+ "findings": [...],
+ "positive_controls": [{"control": "description", "evidence": "where observed", "files": ["file:line"]}]
+}"""
+
+ extraction_semaphore = asyncio.Semaphore(5)
+ all_extracted = {}
+ extraction_errors = []
+
+ async def extract_report(report_key, content):
+ cached = extraction_ns.get(report_key)
+ if cached:
+ print(f" {report_key}: cached ({len(cached.get('findings', []))} findings)")
+ return report_key, cached, None
+
+ async with extraction_semaphore:
+ print(f" {report_key}: extracting...", end=" ", flush=True)
+ messages = [
+ {"role": "user", "content": f"{EXTRACTION_PROMPT_TEMPLATE}\n\nSource report: {report_key}\n\nReport content:\n{content}"}
+ ]
+ try:
+ result, _ = await call_llm(
+ provider=FAST_PROVIDER, model=FAST_MODEL,
+ messages=messages, parameters=FAST_PARAMS, timeout=600,
+ )
+ json_match = re.search(r'\{[\s\S]*\}', result)
+ if json_match:
+ extracted = json.loads(json_match.group())
+ extracted["source_report"] = report_key
+ extraction_ns.set(report_key, extracted)
+ print(f"{len(extracted.get('findings', []))} findings, status: {extracted.get('asvs_status', '?')}")
+ return report_key, extracted, None
+ else:
+ print(f"WARNING: no JSON")
+ return report_key, None, "No JSON in result"
+ except Exception as e:
+ print(f"ERROR: {e}")
+ return report_key, None, str(e)
+
+ extraction_results = await asyncio.gather(*[
+ extract_report(rk, content) for rk, content in reports.items()
+ ])
+ for rk, extracted, error in extraction_results:
+ if extracted:
+ all_extracted[rk] = extracted
+ elif error:
+ extraction_errors.append(rk)
+
+ if extraction_errors:
+ print(f"\nRetrying {len(extraction_errors)} failed extractions...")
+ retry_results = await asyncio.gather(*[
+ extract_report(rk, reports[rk]) for rk in extraction_errors
+ ])
+ for rk, extracted, error in retry_results:
+ if extracted:
+ all_extracted[rk] = extracted
+ extraction_errors.remove(rk)
+
+ total_extracted = sum(len(v.get("findings", [])) for v in all_extracted.values())
+ print(f"\nTotal extracted findings: {total_extracted}")
+
+ # ============================================================
+ # PHASE 2.5: Enrich with ASVS requirement context
+ # ============================================================
+ print("\n=== Loading ASVS requirement context ===")
+
+ asvs_ns = data_store.use_namespace("asvs")
+ ASVS_PREFIX = "asvs:"
+
+ all_asvs_ids = set()
+ for extracted in all_extracted.values():
+ section = extracted.get("asvs_section", "")
+ if section:
+ all_asvs_ids.add(section)
+ for finding in extracted.get("findings", []):
+ s = finding.get("asvs_section", "")
+ if s:
+ all_asvs_ids.add(s)
+
+ asvs_context = {}
+ chapters_cache = {}
+ sections_cache = {}
+
+ for req_id in sorted(all_asvs_ids):
+ req = asvs_ns.get(f"{ASVS_PREFIX}requirements:{req_id}")
+ if not req:
+ continue
+ section_id = req.get("section_id", "")
+ chapter_id = req.get("chapter_id", "")
+ if section_id and section_id not in sections_cache:
+ sections_cache[section_id] = asvs_ns.get(f"{ASVS_PREFIX}sections:{section_id}")
+ if chapter_id and chapter_id not in chapters_cache:
+ chapters_cache[chapter_id] = asvs_ns.get(f"{ASVS_PREFIX}chapters:{chapter_id}")
+ sec = sections_cache.get(section_id) or {}
+ ch = chapters_cache.get(chapter_id) or {}
+ asvs_context[req_id] = {
+ "req_description": req.get("req_description", ""),
+ "level": req.get("level"),
+ "section_name": sec.get("section_name", ""),
+ "section_description": sec.get("description", ""),
+ "chapter_name": ch.get("chapter_name", ""),
+ "control_objective": ch.get("control_objective", ""),
+ }
+
+ print(f" Loaded ASVS context for {len(asvs_context)} requirements")
+
+ def get_asvs_level(section_id):
+ ctx = asvs_context.get(section_id)
+ if ctx and ctx.get("level"):
+ return f"L{ctx['level']}"
+ return "L?"
+
+ # ============================================================
+ # PHASE 3: Domain-Grouped Consolidation
+ # ============================================================
+ print("\n=== PHASE 3: Domain-grouped consolidation ===")
+
+ # Domain groups from input or ATR defaults
+ if domain_groups_raw:
+ DOMAIN_GROUPS = json.loads(domain_groups_raw) if isinstance(domain_groups_raw, str) else domain_groups_raw
+ print(f" Using provided domain groups: {len(DOMAIN_GROUPS)} domains", flush=True)
+ else:
+ DOMAIN_GROUPS = {
+ "input_encoding": ["1.1.1","1.1.2","1.2.1","1.2.2","1.2.3","1.2.4","1.2.5","1.3.1","1.3.2","1.4.1","1.4.2","1.4.3","1.5.1"],
+ "business_logic": ["2.1.1","2.2.1","2.2.2","2.3.1","2.4.1"],
+ "session_csrf": ["3.2.1","3.2.2","3.3.1","3.4.1","3.4.2","3.5.1","3.5.2","3.5.3","3.7.1","3.7.2"],
+ "content_type": ["4.1.1","4.2.1","4.3.1","4.3.2","4.4.1"],
+ "file_path": ["5.1.1","5.2.1","5.2.2","5.3.1","5.3.2","5.4.1","5.4.2","5.4.3"],
+ "auth_rate_limit": ["6.1.1","6.2.1","6.2.2","6.2.3","6.2.4","6.2.5","6.2.6","6.2.7","6.2.8","6.3.1","6.3.2","6.4.1","6.4.2","6.5.1","6.5.2","6.5.3","6.5.4","6.5.5","6.6.1","6.6.2","6.6.3","6.8.1","6.8.2","6.8.3","6.8.4"],
+ "session_token": ["7.1.1","7.1.2","7.1.3","7.2.1","7.2.2","7.2.3","7.2.4","7.3.1","7.3.2","7.4.1","7.4.2","7.5.1","7.5.2","7.6.1","7.6.2"],
+ "authorization": ["8.1.1","8.2.1","8.2.2","8.3.1","8.4.1"],
+ "jwt_token": ["9.1.1","9.1.2","9.1.3","9.2.1"],
+ "oauth": ["10.1.1","10.1.2","10.2.1","10.2.2","10.3.1","10.3.2","10.3.3","10.3.4","10.4.1","10.4.2","10.4.3","10.4.4","10.4.5","10.5.1","10.5.2","10.5.3","10.5.4","10.5.5","10.6.1","10.6.2","10.7.1","10.7.2","10.7.3"],
+ "crypto_tls": ["11.1.1","11.1.2","11.2.1","11.2.2","11.2.3","11.3.1","11.3.2","11.4.1","11.5.1","11.6.1","12.1.1","12.2.1","12.2.2","12.3.1","12.3.2","12.3.3","12.3.4"],
+ "api_scm_client": ["13.1.1","13.2.1","13.2.2","13.2.3","13.2.4","13.2.5","13.3.1","13.3.2","13.4.1","14.1.1","14.1.2","14.2.1","14.3.1"],
+ "dependencies": ["15.1.1","15.2.1","15.3.1"],
+ "audit_logging": ["16.1.1","16.2.1","16.2.2","16.2.3","16.2.4","16.2.5","16.3.1","16.3.2","16.3.3","16.3.4","16.4.1","16.4.2","16.4.3","16.5.1","16.5.2","16.5.3"],
+ "webrtc": ["17.1.1","17.2.1","17.2.2","17.2.3","17.2.4","17.3.1","17.3.2"],
+ }
+ print(f" Using default ATR domain groups: {len(DOMAIN_GROUPS)} domains", flush=True)
+
+ section_to_domain = {}
+ for domain, sections in DOMAIN_GROUPS.items():
+ for section in sections:
+ section_to_domain[section] = domain
+
+ chapter_domain_map = {}
+ for domain, sections in DOMAIN_GROUPS.items():
+ for section in sections:
+ chapter = ".".join(section.split(".")[:2])
+ chapter_domain_map[chapter] = domain
+
+ domain_reports = {domain: {} for domain in DOMAIN_GROUPS}
+ domain_reports["misc"] = {}
+
+ for report_key, extracted in all_extracted.items():
+ asvs_section = extracted.get("asvs_section", "")
+ if not asvs_section:
+ filename_part = report_key.split(":")[-1] if ":" in report_key else report_key
+ match = re.match(r'(\d+\.\d+\.\d+)', filename_part)
+ if match:
+ asvs_section = match.group(1)
+
+ domain = section_to_domain.get(asvs_section)
+ if not domain and asvs_section:
+ chapter = ".".join(asvs_section.split(".")[:2])
+ domain = chapter_domain_map.get(chapter, "misc")
+ if not domain:
+ domain = "misc"
+
+ if domain not in domain_reports:
+ domain_reports[domain] = {}
+ domain_reports[domain][report_key] = extracted
+
+ domain_reports = {d: r for d, r in domain_reports.items() if r}
+
+ for domain, rpts in domain_reports.items():
+ finding_count = sum(len(r.get("findings", [])) for r in rpts.values())
+ print(f" {domain}: {len(rpts)} reports, {finding_count} findings")
+
+ CONSOLIDATION_PROMPT = f"""You are a security audit consolidator. You are given extracted findings from multiple ASVS audit reports within the SAME security domain.
+
+Audit scope: up to {level}
+
+Your job is to:
+1. **Identify TRUE duplicates**: The EXACT same vulnerability in the EXACT same code location. Merge these into ONE finding.
+2. **Preserve EVERY unique finding**. If in doubt, keep findings SEPARATE.
+3. **Track ASVS levels**: Each finding must list which ASVS level(s) apply based on its ASVS sections.
+4. **Use the ASVS requirement descriptions** provided to understand what each section tests for.
+5. **Do NOT add cross-references between findings.**
+
+**Deduplication test**: If a developer could fix one WITHOUT fixing the other, they are SEPARATE findings.
+
+Return valid JSON:
+{{
+ "domain": "domain_name",
+ "consolidated_findings": [
+ {{
+ "temp_id": "DOMAIN-N",
+ "severity": "Critical|High|Medium|Low|Informational",
+ "title": "descriptive title",
+ "description": "full description",
+ "cwe": "CWE-NNN or null",
+ "asvs_sections": ["X.Y.Z", ...],
+ "asvs_levels": ["L1", "L2"],
+ "affected_files": [{{"file": "path", "line": "N"}}],
+ "source_reports": ["filename.md", ...],
+ "recommended_remediation": "specific fix",
+ "merged_from": ["original finding IDs"]
+ }}
+ ],
+ "positive_controls": [
+ {{"control": "description", "evidence": "where observed", "files": ["file:line"]}}
+ ],
+ "asvs_statuses": {{
+ "X.Y.Z": {{"status": "Pass|Fail|Partial|N/A", "title": "section title"}}
+ }},
+ "dedup_log": [
+ "Merged FINDING-X with FINDING-Y because: reason"
+ ]
+}}"""
+
+ consolidation_semaphore = asyncio.Semaphore(3)
+ domain_consolidated = {}
+
+ def build_asvs_context_block(rpts):
+ domain_asvs_sections = set()
+ for rpt_data in rpts.values():
+ s = rpt_data.get("asvs_section", "")
+ if s:
+ domain_asvs_sections.add(s)
+ if not domain_asvs_sections:
+ return ""
+ ctx_lines = []
+ for sid in sorted(domain_asvs_sections):
+ ctx = asvs_context.get(sid)
+ if ctx:
+ ctx_lines.append(
+ f"- **{sid}** ({get_asvs_level(sid)}, {ctx['section_name']}): "
+ f"{ctx['req_description'][:300]}"
+ )
+ if not ctx_lines:
+ return ""
+ return "\n\n## ASVS Requirement Descriptions\n" + "\n".join(ctx_lines)
+
+ async def consolidate_domain(domain, rpts):
+ cached = consolidation_ns.get(domain)
+ if cached:
+ print(f" {domain}: cached ({len(cached.get('consolidated_findings', []))} findings)")
+ return domain, cached
+
+ async with consolidation_semaphore:
+ print(f" {domain}: consolidating {len(rpts)} reports...", flush=True)
+ domain_data = json.dumps(rpts, indent=2, default=str)
+ asvs_block = build_asvs_context_block(rpts)
+ user_msg = f"Domain: {domain}{asvs_block}\n\nExtracted findings:\n\n{domain_data}"
+ messages = [{"role": "user", "content": f"{CONSOLIDATION_PROMPT}\n\n{user_msg}"}]
+
+ msg_tokens = count_message_tokens(messages, FAST_PROVIDER, FAST_MODEL)
+ limit = int(FAST_CONTEXT_WINDOW * 0.80)
+
+ if msg_tokens > limit:
+ items = list(rpts.items())
+ mid = len(items) // 2
+ sub_results = []
+ for si, sub in enumerate([dict(items[:mid]), dict(items[mid:])]):
+ sub_data = json.dumps(sub, indent=2, default=str)
+ sub_asvs = build_asvs_context_block(sub)
+ sub_msg = f"Domain: {domain} (sub-group {si+1}/2){sub_asvs}\n\nFindings:\n\n{sub_data}"
+ sub_messages = [{"role": "user", "content": f"{CONSOLIDATION_PROMPT}\n\n{sub_msg}"}]
+ try:
+ result, _ = await call_llm(
+ provider=FAST_PROVIDER, model=FAST_MODEL,
+ messages=sub_messages,
+ parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=600,
+ )
+ json_match = re.search(r'\{[\s\S]*\}', result)
+ if json_match:
+ sub_results.append(json.loads(json_match.group()))
+ except Exception as e:
+ print(f" ERROR in sub-group {si+1}: {e}")
+ if sub_results:
+ merged = {"domain": domain, "consolidated_findings": [], "positive_controls": [], "asvs_statuses": {}, "dedup_log": []}
+ for sr in sub_results:
+ merged["consolidated_findings"].extend(sr.get("consolidated_findings", []))
+ merged["positive_controls"].extend(sr.get("positive_controls", []))
+ merged["asvs_statuses"].update(sr.get("asvs_statuses", {}))
+ merged["dedup_log"].extend(sr.get("dedup_log", []))
+ consolidation_ns.set(domain, merged)
+ return domain, merged
+ return domain, None
+
+ try:
+ result, _ = await call_llm(
+ provider=FAST_PROVIDER, model=FAST_MODEL,
+ messages=messages,
+ parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=600,
+ )
+ json_match = re.search(r'\{[\s\S]*\}', result)
+ if json_match:
+ consolidated = json.loads(json_match.group())
+ consolidation_ns.set(domain, consolidated)
+ print(f" Result: {len(consolidated.get('consolidated_findings', []))} findings")
+ return domain, consolidated
+ return domain, None
+ except Exception as e:
+ print(f" ERROR: {e}")
+ return domain, None
+
+ consolidation_results = await asyncio.gather(*[
+ consolidate_domain(domain, rpts) for domain, rpts in domain_reports.items()
+ ])
+
+ for domain, consolidated in consolidation_results:
+ if consolidated:
+ domain_consolidated[domain] = consolidated
+ else:
+ rpts = domain_reports[domain]
+ fallback = {"domain": domain, "consolidated_findings": [], "positive_controls": [], "asvs_statuses": {}, "dedup_log": []}
+ for report_key, extracted in rpts.items():
+ for fi, finding in enumerate(extracted.get("findings", [])):
+ asvs_sec = finding.get("asvs_section", "")
+ fallback["consolidated_findings"].append({
+ "temp_id": f"{domain.upper()}-{fi+1}",
+ "severity": finding.get("severity", "Medium"),
+ "title": finding.get("title", "Unknown"),
+ "description": finding.get("description", ""),
+ "cwe": finding.get("cwe"),
+ "asvs_sections": [asvs_sec],
+ "asvs_levels": [get_asvs_level(asvs_sec)],
+ "affected_files": finding.get("affected_files", []),
+ "source_reports": [report_key],
+ "recommended_remediation": finding.get("recommended_remediation", ""),
+ "merged_from": [],
+ })
+ section = extracted.get("asvs_section", "")
+ if section:
+ fallback["asvs_statuses"][section] = {
+ "status": extracted.get("asvs_status", "Unknown"),
+ "title": extracted.get("asvs_section_title", ""),
+ }
+ fallback["positive_controls"].extend(extracted.get("positive_controls", []))
+ domain_consolidated[domain] = fallback
+
+ total_consolidated = sum(len(d.get("consolidated_findings", [])) for d in domain_consolidated.values())
+ print(f"\nTotal consolidated findings (pre-cross-domain): {total_consolidated}")
+
+ # ============================================================
+ # PHASE 3.5: Cross-Domain Deduplication
+ # ============================================================
+ print("\n=== PHASE 3.5: Cross-domain deduplication ===")
+
+ xd_all = []
+ for domain, data in domain_consolidated.items():
+ for fi, finding in enumerate(data.get("consolidated_findings", [])):
+ finding["_xd_domain"] = domain
+ finding["_xd_idx"] = fi
+ xd_all.append(finding)
+
+ def primary_file(finding):
+ af = finding.get("affected_files", [])
+ if not af:
+ return ""
+ first = af[0]
+ if isinstance(first, dict):
+ return first.get("file", "").split(":")[0].split(" (")[0].strip().strip("`")
+ return str(first).split(":")[0].split(" (")[0].strip().strip("`")
+
+ def norm_title(t):
+ t = t.lower().strip()
+ for noise in ["completely ", "critical: ", "high: ", "medium: ", "low: "]:
+ t = t.replace(noise, "")
+ t = re.sub(r'[^a-z0-9 ]', '', t)
+ t = re.sub(r'\s+', ' ', t).strip()
+ return t
+
+ file_groups = {}
+ for finding in xd_all:
+ pf = primary_file(finding)
+ if pf:
+ file_groups.setdefault(pf, []).append(finding)
+
+ xd_merge_count = 0
+ xd_removed = set()
+
+ def merge_into(primary, duplicate):
+ for sr in duplicate.get("source_reports", []):
+ if sr not in primary.get("source_reports", []):
+ primary.setdefault("source_reports", []).append(sr)
+ for sec in duplicate.get("asvs_sections", []):
+ if sec not in primary.get("asvs_sections", []):
+ primary.setdefault("asvs_sections", []).append(sec)
+ for lv in duplicate.get("asvs_levels", []):
+ if lv not in primary.get("asvs_levels", []):
+ primary.setdefault("asvs_levels", []).append(lv)
+ primary.setdefault("merged_from", []).append(duplicate.get("temp_id", "unknown"))
+ if len(duplicate.get("description", "")) > len(primary.get("description", "")):
+ primary["description"] = duplicate["description"]
+ if len(duplicate.get("recommended_remediation", "")) > len(primary.get("recommended_remediation", "")):
+ primary["recommended_remediation"] = duplicate["recommended_remediation"]
+
+ for pf, group in file_groups.items():
+ if len(group) < 2:
+ continue
+ title_clusters = {}
+ for finding in group:
+ nt = norm_title(finding.get("title", ""))
+ title_clusters.setdefault(nt, []).append(finding)
+ for nt, cluster in title_clusters.items():
+ if len(cluster) < 2:
+ continue
+ cluster.sort(key=lambda f: len(f.get("source_reports", [])), reverse=True)
+ primary = cluster[0]
+ for duplicate in cluster[1:]:
+ dup_key = (duplicate["_xd_domain"], duplicate["_xd_idx"])
+ if dup_key in xd_removed:
+ continue
+ merge_into(primary, duplicate)
+ xd_removed.add(dup_key)
+ xd_merge_count += 1
+
+ # LLM-assisted dedup for large file groups
+ xd_llm_groups = {pf: [f for f in group if (f["_xd_domain"], f["_xd_idx"]) not in xd_removed]
+ for pf, group in file_groups.items()}
+ xd_llm_groups = {pf: g for pf, g in xd_llm_groups.items() if len(g) >= 3}
+
+ if xd_llm_groups:
+ XD_DEDUP_PROMPT = """You are deduplicating security findings that affect the SAME file but came from DIFFERENT ASVS domain groups.
+
+Two findings are TRUE DUPLICATES if they describe the EXACT SAME bug and a developer fixing one would fix the other.
+
+Return JSON: {"merges": [{"keep": "TEMP-ID", "absorb": ["TEMP-ID-1"], "reason": "same bug"}]}
+If no duplicates: {"merges": []}"""
+
+ for pf, group in xd_llm_groups.items():
+ group_data = [{"temp_id": f"{f['_xd_domain']}:{f.get('temp_id','?')}", "title": f.get("title",""), "description": f.get("description","")[:500], "asvs_sections": f.get("asvs_sections",[]), "affected_files": f.get("affected_files",[])[:3]} for f in group]
+ messages = [{"role": "user", "content": f"{XD_DEDUP_PROMPT}\n\nFile: {pf}\n\n{json.dumps(group_data, indent=2, default=str)}"}]
+ if count_message_tokens(messages, FAST_PROVIDER, FAST_MODEL) > int(FAST_CONTEXT_WINDOW * 0.60):
+ continue
+ try:
+ result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL, messages=messages, parameters=FAST_PARAMS, timeout=120)
+ json_match = re.search(r'\{[\s\S]*\}', result)
+ if json_match:
+ dedup_result = json.loads(json_match.group())
+ lookup = {f"{f['_xd_domain']}:{f.get('temp_id','?')}": f for f in group}
+ for merge in dedup_result.get("merges", []):
+ keep = lookup.get(merge.get("keep",""))
+ if not keep:
+ continue
+ for abs_key in merge.get("absorb", []):
+ abs_f = lookup.get(abs_key)
+ if abs_f and (abs_f["_xd_domain"], abs_f["_xd_idx"]) not in xd_removed:
+ merge_into(keep, abs_f)
+ xd_removed.add((abs_f["_xd_domain"], abs_f["_xd_idx"]))
+ xd_merge_count += 1
+ except Exception:
+ pass
+
+ if xd_removed:
+ for domain, data in domain_consolidated.items():
+ data["consolidated_findings"] = [f for fi, f in enumerate(data.get("consolidated_findings", [])) if (domain, fi) not in xd_removed]
+ for domain, data in domain_consolidated.items():
+ for f in data.get("consolidated_findings", []):
+ f.pop("_xd_domain", None)
+ f.pop("_xd_idx", None)
+
+ total_after_xd = sum(len(d.get("consolidated_findings", [])) for d in domain_consolidated.values())
+ print(f"Cross-domain dedup: {xd_merge_count} merges, {total_consolidated} → {total_after_xd} findings")
+
+ # ============================================================
+ # PHASE 4: Final Merge and Report Generation
+ # ============================================================
+ print("\n=== PHASE 4: Final merge and report generation ===")
+
+ severity_order = {"Critical": 0, "High": 1, "Medium": 2, "Low": 3, "Informational": 4}
+ all_findings = []
+
+ for domain, data in domain_consolidated.items():
+ for finding in data.get("consolidated_findings", []):
+ finding["_domain"] = domain
+ if "asvs_levels" not in finding or not finding["asvs_levels"]:
+ levels = set()
+ for sec in finding.get("asvs_sections", []):
+ levels.add(get_asvs_level(sec))
+ finding["asvs_levels"] = sorted(levels) if levels else [level]
+ all_findings.append(finding)
+
+ all_findings.sort(key=lambda f: severity_order.get(f.get("severity", "Informational"), 4))
+
+ for i, finding in enumerate(all_findings, 1):
+ finding["global_id"] = f"FINDING-{i:03d}"
+
+ # Cross-references
+ print("Building cross-references...")
+
+ def extract_primary_file(finding):
+ af = finding.get("affected_files", [])
+ if not af:
+ return ""
+ first = af[0]
+ raw = first.get("file", "") if isinstance(first, dict) else str(first)
+ return re.sub(r'[:\s(].*', '', raw).strip().strip("`")
+
+ def extract_function_names(finding):
+ names = set()
+ for af in finding.get("affected_files", []):
+ raw = af.get("file", "") if isinstance(af, dict) else str(af)
+ for m in re.finditer(r'(?:^|[:\s])([a-zA-Z_]\w*(?:\.\w+)*)\s*\(', raw):
+ names.add(m.group(1))
+ return names
+
+ by_file, by_cwe, by_func = {}, {}, {}
+ for finding in all_findings:
+ gid = finding["global_id"]
+ pf = extract_primary_file(finding)
+ if pf:
+ by_file.setdefault(pf, set()).add(gid)
+ cwe = finding.get("cwe", "")
+ if cwe and cwe != "null":
+ by_cwe.setdefault(cwe, set()).add(gid)
+ for fn in extract_function_names(finding):
+ by_func.setdefault(fn, set()).add(gid)
+
+ for finding in all_findings:
+ gid = finding["global_id"]
+ related = set()
+ cwe = finding.get("cwe", "")
+ if cwe == "null":
+ cwe = ""
+ if cwe and cwe in by_cwe:
+ related |= by_cwe[cwe]
+ for fn in extract_function_names(finding):
+ if fn in by_func:
+ related |= by_func[fn]
+ pf = extract_primary_file(finding)
+ if pf and pf in by_file:
+ for oid in by_file[pf]:
+ if oid == gid:
+ continue
+ other = next((f for f in all_findings if f["global_id"] == oid), None)
+ if not other:
+ continue
+ ocwe = other.get("cwe", "")
+ if ocwe == "null":
+ ocwe = ""
+ if (cwe and ocwe and cwe == ocwe) or (extract_function_names(finding) & extract_function_names(other)):
+ related.add(oid)
+ related.discard(gid)
+ finding["related_findings"] = sorted(related)[:10] if related else []
+
+ all_asvs_statuses = {}
+ for domain, data in domain_consolidated.items():
+ for section, info in data.get("asvs_statuses", {}).items():
+ all_asvs_statuses[section] = info
+ for rk, extracted in all_extracted.items():
+ section = extracted.get("asvs_section", "")
+ if section and section not in all_asvs_statuses:
+ all_asvs_statuses[section] = {
+ "status": extracted.get("asvs_status", "Unknown"),
+ "title": extracted.get("asvs_section_title", ""),
+ }
+
+ all_positive_controls = []
+ for domain, data in domain_consolidated.items():
+ for ctrl in data.get("positive_controls", []):
+ ctrl["_domain"] = domain
+ all_positive_controls.append(ctrl)
+
+ severity_counts = {"Critical": 0, "High": 0, "Medium": 0, "Low": 0, "Informational": 0}
+ for f in all_findings:
+ sev = f.get("severity", "Informational")
+ severity_counts[sev] = severity_counts.get(sev, 0) + 1
+
+ commit_info = "N/A"
+ for part in directories[0].split("/"):
+ if len(part) >= 7 and re.match(r'^[0-9a-f]+$', part):
+ commit_info = part
+ break
+
+ findings_by_severity = {}
+ for f in all_findings:
+ findings_by_severity.setdefault(f.get("severity", "Informational"), []).append(f)
+
+ # Executive summary (Opus)
+ print(f"Generating executive summary...")
+ exec_summary_prompt = f"""Generate the opening sections of a security audit consolidated report in Markdown.
+
+Audit scope: up to {level}
+
+Repository: {owner}/{repo}
+Directories: {', '.join(directories)}
+ASVS Level: {level}
+Commit: {commit_info}
+Date: {audit_date}
+Auditor: Tooling Agents
+Source reports: {total_reports}
+Total findings: {len(all_findings)}
+
+Severity: Critical={severity_counts.get('Critical',0)}, High={severity_counts.get('High',0)}, Medium={severity_counts.get('Medium',0)}, Low={severity_counts.get('Low',0)}, Info={severity_counts.get('Informational',0)}
+
+Finding titles:
+"""
+ for f in all_findings:
+ lvs = ", ".join(f.get("asvs_levels", []))
+ exec_summary_prompt += f"- [{f.get('severity')}] [{lvs}] {f.get('global_id')}: {f.get('title')} (ASVS: {', '.join(f.get('asvs_sections', []))})\n"
+
+ exec_summary_prompt += f"""
+Positive controls: {json.dumps(all_positive_controls[:50], indent=2, default=str)}
+
+Generate ONLY:
+1. Report Metadata table (Repository, ASVS Level, Commit, Date, Auditor, Source Reports, Total Findings)
+2. Executive Summary with severity distribution, level coverage, top 5 risks, positive controls
+
+End with ---."""
+
+ exec_result, _ = await call_llm(provider=HEAVY_PROVIDER, model=HEAVY_MODEL,
+ messages=[{"role": "user", "content": exec_summary_prompt}],
+ parameters={**HEAVY_PARAMS, "max_tokens": 32000}, timeout=900)
+ exec_result = sanitize_md_html(exec_result)
+
+ # Findings (Sonnet, batched)
+ FINDING_FORMAT = """For each finding: #### FINDING-NNN: Title, attribute table (Severity, ASVS Level(s), CWE, ASVS sections, Files, Source Reports, Related), Description, Remediation. Use emojis: 🔴 Critical, 🟠 High, 🟡 Medium, 🔵 Low, ⚪ Info. Separate with ---."""
+
+ findings_md_parts = []
+ MAX_PER_BATCH = 30
+ sev_section_names = {"Critical": "3.1 Critical", "High": "3.2 High", "Medium": "3.3 Medium", "Low": "3.4 Low", "Informational": "3.5 Informational"}
+
+ for sev in ["Critical", "High", "Medium", "Low", "Informational"]:
+ sev_findings = findings_by_severity.get(sev, [])
+ if not sev_findings:
+ continue
+ for sb_idx in range(0, len(sev_findings), MAX_PER_BATCH):
+ batch = sev_findings[sb_idx:sb_idx + MAX_PER_BATCH]
+ is_first = (sb_idx == 0)
+ prompt = f"""Format these {len(batch)} {sev} findings into Markdown.\n{FINDING_FORMAT}\n{"Start with: ### " + sev_section_names[sev] if is_first else "Continue. No header."}\n\n{json.dumps(batch, indent=2, default=str)}\n\nGenerate ALL {len(batch)} findings with full detail."""
+ for attempt in range(3):
+ try:
+ result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL,
+ messages=[{"role": "user", "content": prompt}],
+ parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=900)
+ findings_md_parts.append(sanitize_md_html(result))
+ break
+ except Exception as e:
+ if attempt < 2:
+ await asyncio.sleep(5)
+
+ # Tail sections (Sonnet)
+ tail_prompt = f"""Generate final sections:\n## 4. Positive Security Controls (table)\n## 5. ASVS Compliance Summary (table with status)\n## 6. Cross-Reference Matrix\n\nPositive controls: {json.dumps(all_positive_controls[:100], indent=2, default=str)}\nASVS statuses: {json.dumps(all_asvs_statuses, indent=2, default=str)}\nFindings: {json.dumps([{"id": f["global_id"], "sev": f["severity"], "title": f["title"], "asvs": f.get("asvs_sections",[])} for f in all_findings], indent=2, d [...]
+
+ try:
+ tail_result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL,
+ messages=[{"role": "user", "content": tail_prompt}],
+ parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=900)
+ tail_result = sanitize_md_html(tail_result)
+ except:
+ tail_result = "\n*Tail sections generation failed.*\n"
+
+ # Assemble
+ consolidated_md = exec_result.rstrip() + "\n\n## 3. Findings\n\n" + "\n\n".join(findings_md_parts) + "\n\n---\n\n" + tail_result
+
+ # Section 7: Level Coverage (deterministic)
+ s7 = ["\n\n## 7. Level Coverage Analysis\n", f"\n**Audit scope:** up to {level}\n"]
+ s7.append("| Level | Sections Audited | Findings Found |")
+ s7.append("|-------|-----------------|----------------|")
+ for lv_name, lv_num in [("L1", 1), ("L2", 2), ("L3", 3)]:
+ if lv_num > max_level_num:
+ continue
+ lv_sections = sum(1 for rk, ext in all_extracted.items() if get_asvs_level(ext.get("asvs_section", "")) == lv_name)
+ lv_findings = [f for f in all_findings if lv_name in f.get("asvs_levels", [])]
+ s7.append(f"| {lv_name} | {lv_sections} | {len(lv_findings)} |")
+ s7.append(f"\n**Total consolidated findings: {len(all_findings)}**")
+ s7.append("\n*End of Consolidated Security Audit Report*")
+ consolidated_md += "\n".join(s7)
+
+ # Issues file
+ informational_ids = set(f["global_id"] for f in all_findings if "informational" in f.get("severity","").lower())
+ actionable = [f for f in all_findings if f["global_id"] not in informational_ids]
+
+ ISSUE_FMT = """---\n## Issue: FINDING-NNN - Title\n**Labels:** bug, security, priority:sev\n**Description:**\n### Summary\n### Details\n### Remediation\n### Acceptance Criteria\n- [ ] Fixed\n- [ ] Test added\n### References\n### Priority"""
+
+ issues_parts = []
+ for bi in range(0, len(actionable), 75):
+ batch = actionable[bi:bi+75]
+ is_first = (bi == 0)
+ prompt = f"""Generate GitHub issues for {len(batch)} findings.\n{ISSUE_FMT}\n{"Start with: # Security Issues" if is_first else "Continue. No header."}\n\n{json.dumps(batch, indent=2, default=str)}"""
+ for attempt in range(3):
+ try:
+ result, _ = await call_llm(provider=FAST_PROVIDER, model=FAST_MODEL,
+ messages=[{"role": "user", "content": prompt}],
+ parameters={**FAST_PARAMS, "max_tokens": 64000}, timeout=900)
+ issues_parts.append(sanitize_md_html(result))
+ break
+ except:
+ if attempt < 2:
+ await asyncio.sleep(5)
+ else:
+ # Fallback
+ fb = []
+ for f in batch:
+ fb.append(f"---\n\n## Issue: {f['global_id']} - {sanitize_md_html(f.get('title',''))}\n\n**Labels:** bug, security, priority:{f.get('severity','Medium').lower()}\n\n**Description:**\n\n{sanitize_md_html(f.get('description',''))}\n\n**Remediation:** {sanitize_md_html(f.get('recommended_remediation',''))}\n\n**Priority:** {f.get('severity','Medium')}")
+ issues_parts.append("\n\n".join(fb))
+
+ issues_md = "\n\n".join(issues_parts)
+
+ # Quality checks
+ print(f"\n=== Quality Checks ===")
+ finding_sections = set(re.findall(r'#### FINDING-(\d{3})', consolidated_md))
+ print(f"Finding sections: {len(finding_sections)} (expected: {len(all_findings)})")
+ issue_headers = set(re.findall(r'## Issue: (FINDING-\d{3})', issues_md))
+ print(f"Issues: {len(issue_headers)} (expected: {len(actionable)})")
+
+ # Push to GitHub
+ print("\n=== Pushing files ===")
+
+ async def push_file(path, content_str, message):
+ existing_sha = None
+ check = await http_client.get(f"{GITHUB_API}/repos/{owner}/{repo}/contents/{path}", headers=headers, params={"ref": default_branch})
+ if check.status_code == 200:
+ existing_sha = check.json().get("sha")
+ payload = {"message": message, "content": base64.b64encode(content_str.encode("utf-8")).decode("ascii"), "branch": default_branch}
+ if existing_sha:
+ payload["sha"] = existing_sha
+ resp = await http_client.put(f"{GITHUB_API}/repos/{owner}/{repo}/contents/{path}", headers=headers, json=payload)
+ print(f" {'OK' if resp.status_code in (200,201) else 'ERROR'}: {path}")
+
+ await push_file(f"{output_directory}/{consolidated_filename}", consolidated_md, f"Add consolidated audit report ({level})")
+ await push_file(f"{output_directory}/{issues_filename}", issues_md, f"Add security issues ({level})")
+
+ print(f"\n=== Done ===")
+ print(f"Total findings: {len(all_findings)}, Actionable issues: {len(actionable)}")
+
+ return {
+ "outputText": f"Consolidated report and issues pushed.\n"
+ f"Repository: {owner}/{repo}\n"
+ f"Level: {level}\n"
+ f"Findings: {len(all_findings)}\n"
+ f"Files: {output_directory}/{consolidated_filename}, {output_directory}/{issues_filename}"
+ }
+ finally:
+ await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/discover_codebase_architecture.py b/repos/apache/ASVS/agents/discover_codebase_architecture.py
new file mode 100644
index 0000000..6c48b5d
--- /dev/null
+++ b/repos/apache/ASVS/agents/discover_codebase_architecture.py
@@ -0,0 +1,366 @@
+# discover_codebase_architecture
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+ mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+ http_client = httpx.AsyncClient()
+ try:
+ import json
+ import re
+
+ input_namespace = input_dict.get("inputNamespace", "")
+
+ namespaces = [ns.strip() for ns in input_namespace.split(",") if ns.strip()]
+ if not namespaces:
+ all_ns = data_store.list_namespaces()
+ file_ns = [ns for ns in all_ns if ns.startswith("files:")]
+ if file_ns:
+ namespaces = file_ns
+
+ if not namespaces:
+ return {"outputText": json.dumps({"error": f"No namespaces provided. Available: {data_store.list_namespaces()}"})}
+
+ repo_name = "unknown"
+ for ns in namespaces:
+ if ns.startswith("files:"):
+ repo_name = ns.replace("files:", "")
+ break
+
+ print(f"Discovering architecture for: {repo_name}", flush=True)
+ print(f"Namespaces: {namespaces}", flush=True)
+
+ # =============================================================
+ # Model configuration
+ # =============================================================
+ PROVIDER = "bedrock"
+ MODEL = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+ PARAMS = {"temperature": 0.7, "max_tokens": 16384}
+ CONTEXT_WINDOW = get_context_window(PROVIDER, MODEL)
+ SAFE_LIMIT = int(CONTEXT_WINDOW * 0.40)
+
+ # =============================================================
+ # Step 1: Read file paths and previews
+ # =============================================================
+ print("\n=== Step 1: Reading file paths and previews ===", flush=True)
+
+ SKIP_DIRS = {
+ 'node_modules', 'vendor', 'third_party', 'third-party',
+ 'dist', 'build', 'out', 'target',
+ '__pycache__', '.pytest_cache', '.mypy_cache', 'coverage', '.next', '.nuxt',
+ 'assets', 'images', 'img', 'static/images', 'static/fonts', 'static/webfonts',
+ 'public/images', 'fonts', 'webfonts',
+ '.github/workflows',
+ 'venv', '.venv', 'env', '.env',
+ '.git', '.idea', '.vscode',
+ }
+ SKIP_FILES = {
+ 'package-lock.json', 'yarn.lock', 'poetry.lock', 'Cargo.lock',
+ 'composer.lock', 'pnpm-lock.yaml', 'Gemfile.lock', 'uv.lock',
+ 'LICENSE', 'LICENSE.md', 'LICENSE.txt',
+ 'README.md', 'README.rst', 'README.txt', 'README',
+ 'CHANGELOG.md', 'CHANGELOG', 'CONTRIBUTING.md', 'CODE_OF_CONDUCT.md',
+ '.gitignore', '.dockerignore', '.prettierrc', '.eslintrc', '.editorconfig',
+ '.npmrc', '.yarnrc',
+ }
+ SKIP_EXTENSIONS = {
+ '.min.js', '.min.css', '.bundle.js', '.bundle.css',
+ '.map', '.css', '.scss', '.less', '.sass', '.styl',
+ '.woff', '.woff2', '.ttf', '.eot', '.otf',
+ '.png', '.jpg', '.jpeg', '.gif', '.svg', '.ico', '.webp', '.bmp',
+ '.mp3', '.mp4', '.wav', '.avi', '.mov', '.webm', '.ogg',
+ '.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx',
+ '.zip', '.tar', '.gz', '.rar', '.7z', '.lock',
+ '.exe', '.dll', '.so', '.dylib', '.pyc', '.pyo', '.class', '.o', '.obj',
+ }
+
+ def should_skip_file(filepath):
+ path_lower = filepath.lower()
+ parts = filepath.split('/')
+ for part in parts[:-1]:
+ if part.lower() in SKIP_DIRS:
+ return True
+ filename = parts[-1] if parts else ''
+ if filename in SKIP_FILES:
+ return True
+ for ext in SKIP_EXTENSIONS:
+ if path_lower.endswith(ext):
+ return True
+ return False
+
+ all_files = {}
+ for ns in namespaces:
+ ns_store = data_store.use_namespace(ns)
+ keys = ns_store.list_keys()
+ file_contents = ns_store.get_many(keys) if keys else {}
+ for k, v in file_contents.items():
+ if v is not None and not should_skip_file(k):
+ content = v if isinstance(v, str) else json.dumps(v, default=str) if v else ""
+ all_files[k] = content
+
+ print(f"Total code files: {len(all_files)}", flush=True)
+
+ if not all_files:
+ return {"outputText": json.dumps({"error": f"No code files found in namespaces {namespaces}"})}
+
+ file_previews = {}
+ for path, content in all_files.items():
+ lines = content.split('\n')
+ file_previews[path] = '\n'.join(lines[:30])
+
+ total_lines = sum(len(content.split('\n')) for content in all_files.values())
+ print(f"Total lines: {total_lines}", flush=True)
+
+ # =============================================================
+ # Step 2: Architecture classification (Sonnet, batched)
+ # =============================================================
+ print("\n=== Step 2: Architecture classification ===", flush=True)
+
+ CLASSIFY_PROMPT = """You are a security architect analyzing a codebase structure.
+
+Given the file paths and code previews below, identify the codebase's security architecture.
+
+Return ONLY a JSON object with this structure:
+{
+ "framework": "e.g., FastAPI, Django, Flask, Express, Spring",
+ "language": "e.g., Python, Java, JavaScript",
+ "auth_systems": [
+ {"name": "description", "files": ["path1", "path2"]}
+ ],
+ "api_layers": [
+ {"name": "description", "files": ["path1", "path2"]}
+ ],
+ "data_layer": {
+ "database": "e.g., SQLAlchemy, Django ORM",
+ "encryption": "e.g., Fernet, bcrypt",
+ "secrets": "e.g., Vault, env vars, config file",
+ "files": ["path1", "path2"]
+ },
+ "execution_model": {
+ "description": "e.g., async web server, task workers, DAG executor",
+ "files": ["path1", "path2"]
+ },
+ "security_relevant_areas": [
+ {"area": "short name", "description": "what it does", "files": ["path1", "path2"]}
+ ],
+ "trust_model": "one paragraph describing who is trusted and what boundaries exist"
+}
+"""
+ template_tokens = count_tokens(CLASSIFY_PROMPT, PROVIDER, MODEL)
+ preview_budget = SAFE_LIMIT - template_tokens
+
+ path_list = "\n".join(sorted(all_files.keys()))
+ path_tokens = count_tokens(path_list, PROVIDER, MODEL)
+
+ if path_tokens + template_tokens < SAFE_LIMIT:
+ entries = []
+ current_tokens = 0
+ for path in sorted(all_files.keys()):
+ entry = f"\n--- {path} ---\n{file_previews[path]}\n"
+ entry_tokens = count_tokens(entry, PROVIDER, MODEL)
+ if current_tokens + entry_tokens > preview_budget:
+ break
+ entries.append(entry)
+ current_tokens += entry_tokens
+ classify_content = CLASSIFY_PROMPT + "\nFILES:\n" + "".join(entries)
+ else:
+ classify_content = CLASSIFY_PROMPT + f"\nFILE PATHS ({len(all_files)} files):\n" + path_list
+
+ print(f" Classification prompt: {count_tokens(classify_content, PROVIDER, MODEL)} tokens", flush=True)
+
+ architecture = {}
+ for attempt in range(2):
+ try:
+ result, _ = await call_llm(
+ provider=PROVIDER, model=MODEL,
+ messages=[{"role": "user", "content": classify_content}],
+ parameters=PARAMS,
+ timeout=300,
+ )
+ json_match = re.search(r'\{[\s\S]*\}', result)
+ if json_match:
+ architecture = json.loads(json_match.group())
+ print(f" Framework: {architecture.get('framework', '?')}", flush=True)
+ print(f" Auth systems: {len(architecture.get('auth_systems', []))}", flush=True)
+ print(f" Security areas: {len(architecture.get('security_relevant_areas', []))}", flush=True)
+ break
+ except Exception as e:
+ if attempt == 0:
+ print(f" Attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+ await asyncio.sleep(5)
+ else:
+ print(f" Classification FAILED: {e}", flush=True)
+
+ if not architecture:
+ return {"outputText": json.dumps({"error": "Failed to classify codebase architecture"})}
+
+ # =============================================================
+ # Step 3: Generate security domains (Sonnet)
+ # =============================================================
+ print("\n=== Step 3: Generating security domains ===", flush=True)
+
+ asvs_sections_available = []
+ try:
+ asvs_ns = data_store.use_namespace("asvs")
+ all_keys = asvs_ns.list_keys()
+ req_keys = [k for k in all_keys if k.startswith("asvs:requirements:")]
+ for rk in sorted(req_keys):
+ req = asvs_ns.get(rk)
+ if req:
+ section_id = rk.replace("asvs:requirements:", "")
+ req_level = req.get("level", "?")
+ desc = req.get("req_description", "")[:100]
+ asvs_sections_available.append(f"{section_id} (L{req_level}): {desc}")
+ except Exception as e:
+ print(f" WARNING: Could not load ASVS sections: {e}", flush=True)
+
+ asvs_list = "\n".join(asvs_sections_available[:200])
+
+ DOMAIN_PROMPT = f"""Based on this codebase architecture, generate security audit domains.
+
+## Codebase Architecture
+{json.dumps(architecture, indent=2, default=str)}
+
+## Available ASVS Sections
+{asvs_list}
+
+## Instructions
+
+Generate security domains that reflect THIS codebase's actual architecture.
+Each domain should:
+1. Have a short snake_case name
+2. Map to specific ASVS sections from the list above
+3. List the files that belong to it (use exact paths from the architecture data)
+4. Include a context paragraph that an auditor needs to understand this domain
+
+Group ASVS sections by the code area they'd be testing, NOT by ASVS chapter number.
+Every ASVS section should appear in exactly one domain.
+
+Return ONLY a JSON object:
+{{
+ "domains": [
+ {{
+ "name": "snake_case_name",
+ "description": "what this domain covers",
+ "asvs_sections": ["X.Y.Z", ...],
+ "files": ["path/to/file.py", ...],
+ "context": "paragraph explaining the architecture of this domain for auditors"
+ }}
+ ]
+}}"""
+
+ domains = []
+ for attempt in range(2):
+ try:
+ result, _ = await call_llm(
+ provider=PROVIDER, model=MODEL,
+ messages=[{"role": "user", "content": DOMAIN_PROMPT}],
+ parameters={**PARAMS, "max_tokens": 32000},
+ timeout=300,
+ )
+ json_match = re.search(r'\{[\s\S]*\}', result)
+ if json_match:
+ domain_result = json.loads(json_match.group())
+ domains = domain_result.get("domains", [])
+ print(f" Generated {len(domains)} domains", flush=True)
+ for d in domains:
+ print(f" {d['name']}: {len(d.get('asvs_sections', []))} sections, {len(d.get('files', []))} files", flush=True)
+ break
+ except Exception as e:
+ if attempt == 0:
+ print(f" Attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+ await asyncio.sleep(5)
+ else:
+ print(f" Domain generation FAILED: {e}", flush=True)
+
+ if not domains:
+ return {"outputText": json.dumps({"error": "Failed to generate security domains"})}
+
+ for domain in domains:
+ line_count = 0
+ for path in domain.get("files", []):
+ if path in all_files:
+ line_count += len(all_files[path].split('\n'))
+ domain["estimated_lines"] = line_count
+
+ # =============================================================
+ # Step 4: Generate false positive guidance (Sonnet)
+ # =============================================================
+ print("\n=== Step 4: Generating false positive guidance ===", flush=True)
+
+ FP_PROMPT = f"""Based on this codebase architecture, identify patterns that an ASVS security auditor would INCORRECTLY flag as vulnerabilities.
+
+## Codebase Architecture
+{json.dumps(architecture, indent=2, default=str)}
+
+For each false positive pattern, write a concise statement:
+"[What auditor would flag] is intentional because [reason] — auditors should focus on [what matters instead]"
+
+Return ONLY a JSON array of strings:
+[
+ "Pattern X is intentional because Y — auditors should focus on Z instead",
+ ...
+]"""
+
+ false_positive_guidance = []
+ try:
+ result, _ = await call_llm(
+ provider=PROVIDER, model=MODEL,
+ messages=[{"role": "user", "content": FP_PROMPT}],
+ parameters=PARAMS,
+ timeout=120,
+ )
+ json_match = re.search(r'\[[\s\S]*\]', result)
+ if json_match:
+ false_positive_guidance = json.loads(json_match.group())
+ print(f" Generated {len(false_positive_guidance)} patterns", flush=True)
+ except Exception as e:
+ print(f" False positive generation failed ({type(e).__name__}), continuing without", flush=True)
+
+ # =============================================================
+ # Step 5: Assemble output
+ # =============================================================
+ domain_groups = {}
+ for domain in domains:
+ domain_groups[domain["name"]] = domain.get("asvs_sections", [])
+
+ passes = []
+ for domain in domains:
+ passes.append({
+ "name": domain["name"],
+ "description": domain.get("description", ""),
+ "asvs_sections": domain.get("asvs_sections", []),
+ "files": domain.get("files", []),
+ "domain_context": domain.get("context", ""),
+ "estimated_lines": domain.get("estimated_lines", 0),
+ })
+ passes.sort(key=lambda p: p["estimated_lines"])
+
+ pass_config = {
+ "repository": repo_name,
+ "architecture_summary": architecture.get("trust_model", ""),
+ "framework": architecture.get("framework", ""),
+ "language": architecture.get("language", ""),
+ "total_files": len(all_files),
+ "total_lines": total_lines,
+ "passes": passes,
+ "domain_groups": domain_groups,
+ "false_positive_guidance": false_positive_guidance,
+ }
+
+ discovery_ns = data_store.use_namespace(f"discovery:{repo_name}")
+ discovery_ns.set("pass_config", pass_config)
+ discovery_ns.set("architecture", architecture)
+
+ print(f"\n=== Discovery complete ===", flush=True)
+ print(f"Repository: {repo_name}", flush=True)
+ print(f"Domains: {len(domains)}", flush=True)
+ for p in passes:
+ print(f" {p['name']}: {len(p['asvs_sections'])} sections, {p['estimated_lines']} lines", flush=True)
+
+ return {"outputText": json.dumps(pass_config, indent=2, default=str)}
+
+ finally:
+ await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/download_github_repo_to_data_store.py b/repos/apache/ASVS/agents/download_github_repo_to_data_store.py
new file mode 100644
index 0000000..fa3576f
--- /dev/null
+++ b/repos/apache/ASVS/agents/download_github_repo_to_data_store.py
@@ -0,0 +1,149 @@
+# download_github_repo_to_datastore
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+ mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+ http_client = httpx.AsyncClient()
+ try:
+ input_text = input_dict.get("inputText", "")
+ lines = input_text.strip().split("\n")
+
+ repo = None
+ token = None
+ path_prefix = ""
+
+ for line in lines:
+ line = line.strip()
+ if not line:
+ continue
+ if line.startswith("ghp_") or line.startswith("github_pat_") or (len(line) > 30 and line.isalnum()):
+ token = line
+ elif "/" in line and repo is None:
+ raw = line.split()[0].strip("/")
+ parts = raw.split("/")
+ if len(parts) >= 2:
+ repo = f"{parts[0]}/{parts[1]}"
+ if len(parts) > 2:
+ path_prefix = "/".join(parts[2:])
+
+ if not repo:
+ return {"outputText": "Error: Could not find a valid repo in format 'owner/repo' in the input."}
+
+ headers = {
+ "Accept": "application/vnd.github+json",
+ "X-GitHub-Api-Version": "2022-11-28"
+ }
+ if token:
+ headers["Authorization"] = f"Bearer {token}"
+
+ repo_url = f"https://api.github.com/repos/{repo}"
+ response = await http_client.get(repo_url, headers=headers)
+
+ if response.status_code != 200:
+ return {"outputText": f"Error fetching repo info: {response.status_code} - repo='{repo}' token_present={token is not None} - {response.text}"}
+
+ repo_info = response.json()
+ default_branch = repo_info.get("default_branch", "main")
+
+ tree_url = f"https://api.github.com/repos/{repo}/git/trees/{default_branch}?recursive=1"
+ response = await http_client.get(tree_url, headers=headers)
+
+ if response.status_code != 200:
+ return {"outputText": f"Error fetching repo tree: {response.status_code} - {response.text}"}
+
+ tree_data = response.json()
+ tree_items = tree_data.get("tree", [])
+ files_to_fetch = [item for item in tree_items if item.get("type") == "blob"]
+
+ # Filter by path prefix if provided
+ if path_prefix:
+ files_to_fetch = [item for item in files_to_fetch if item.get("path", "").startswith(path_prefix + "/") or item.get("path", "") == path_prefix]
+ print(f"Path prefix '{path_prefix}': {len(files_to_fetch)} files match", flush=True)
+
+ files_ns = data_store.use_namespace(f"files:{repo}")
+
+ fetched_count = 0
+ skipped_binary = 0
+ skipped_vendor = 0
+ skipped_large = 0
+ error_count = 0
+ errors = []
+
+ VENDOR_DIRS = {
+ 'node_modules', 'vendor', 'third_party', 'third-party',
+ '.git',
+ }
+
+ MAX_FILE_SIZE = 1_000_000
+
+ import base64
+
+ for item in files_to_fetch:
+ file_path = item.get("path", "")
+ file_size = item.get("size", 0)
+
+ parts = file_path.split("/")
+ if any(part.lower() in VENDOR_DIRS for part in parts[:-1]):
+ skipped_vendor += 1
+ continue
+
+ if file_size > MAX_FILE_SIZE:
+ skipped_large += 1
+ continue
+
+ try:
+ content_url = f"https://api.github.com/repos/{repo}/contents/{file_path}?ref={default_branch}"
+ file_response = await http_client.get(content_url, headers=headers)
+
+ if file_response.status_code == 200:
+ file_data = file_response.json()
+ content_b64 = file_data.get("content", "")
+
+ if content_b64:
+ try:
+ content = base64.b64decode(content_b64).decode("utf-8")
+ files_ns.set(file_path, content)
+ fetched_count += 1
+ except UnicodeDecodeError:
+ skipped_binary += 1
+ else:
+ skipped_binary += 1
+ else:
+ error_count += 1
+ if len(errors) < 10:
+ errors.append(f"{file_path}: {file_response.status_code}")
+ except Exception as e:
+ error_count += 1
+ if len(errors) < 10:
+ errors.append(f"{file_path}: {str(e)}")
+
+ summary_lines = [
+ f"Repository: {repo}",
+ f"Default branch: {default_branch}",
+ ]
+ if path_prefix:
+ summary_lines.append(f"Path prefix: {path_prefix}")
+ summary_lines += [
+ f"Total files in scope: {len(files_to_fetch)}",
+ f"Files fetched and stored: {fetched_count}",
+ f"Skipped (binary): {skipped_binary}",
+ f"Skipped (vendor/third-party dirs): {skipped_vendor}",
+ f"Skipped (>1MB): {skipped_large}",
+ f"Errors: {error_count}",
+ f"",
+ f"Data stored in namespace: files:{repo}",
+ ]
+
+ if errors:
+ summary_lines.append("")
+ summary_lines.append("First few errors:")
+ for err in errors:
+ summary_lines.append(f" - {err}")
+
+ return {"outputText": "\n".join(summary_lines)}
+
+ finally:
+ await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/orchestrate_asvs_audit_to_github.py b/repos/apache/ASVS/agents/orchestrate_asvs_audit_to_github.py
new file mode 100644
index 0000000..52ed339
--- /dev/null
+++ b/repos/apache/ASVS/agents/orchestrate_asvs_audit_to_github.py
@@ -0,0 +1,368 @@
+# orchestrate_asvs_audit_to_github
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+ mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+ http_client = httpx.AsyncClient()
+ try:
+ import json
+
+ # =============================================================
+ # Parse inputs
+ # =============================================================
+ source_repo = input_dict.get("sourceRepo", "") # e.g., "apache/airflow" or "apache/airflow/airflow-core/src"
+ source_token = input_dict.get("sourceToken", "") # PAT for private source repos
+ supplemental_data = input_dict.get("supplementalData", "") # extra namespaces, comma-separated
+ output_repo = input_dict.get("outputRepo", "")
+ output_token = input_dict.get("outputToken", "")
+ output_directory = input_dict.get("outputDirectory", "")
+ discover = input_dict.get("discover", "true")
+ severity_threshold = input_dict.get("severityThreshold", "")
+ consolidate = input_dict.get("consolidate", "true")
+ level = input_dict.get("level", "")
+
+ if isinstance(discover, str):
+ discover = discover.lower() in ("true", "1", "yes")
+ if isinstance(consolidate, str):
+ consolidate = consolidate.lower() in ("true", "1", "yes")
+
+ level = level.strip().upper()
+ if level and not level.startswith("L"):
+ level = f"L{level}"
+ LEVEL_ORDER = {"L1": 1, "L2": 2, "L3": 3}
+ max_level_num = LEVEL_ORDER.get(level, 3)
+
+ if not source_repo:
+ return {"outputText": "Error: sourceRepo is required (e.g., 'apache/airflow', 'apache/airflow/src', or 'https://github.com/apache/airflow/tree/main/src')"}
+
+ # Derive repo name, path prefix, and namespace from sourceRepo
+ # Accepts:
+ # "apache/steve" → repo=apache/steve, path=""
+ # "apache/steve/v3" → repo=apache/steve, path="v3"
+ # "apache/airflow/airflow-core/src" → repo=apache/airflow, path="airflow-core/src"
+ # "https://github.com/apache/steve/tree/trunk/v3" → repo=apache/steve, path="v3"
+ import re as _re
+ _source = source_repo.strip().strip("/")
+
+ # Strip GitHub URL prefix and tree/branch segment if present
+ _gh_match = _re.match(
+ r'(?:https?://)?github\.com/([^/]+)/([^/]+?)(?:\.git)?(?:/tree/[^/]+(?:/(.+))?)?$',
+ _source,
+ )
+ if _gh_match:
+ repo_owner_name = f"{_gh_match.group(1)}/{_gh_match.group(2)}"
+ repo_short_name = _gh_match.group(2)
+ source_path_prefix = _gh_match.group(3) or ""
+ else:
+ source_parts = _source.split("/")
+ if len(source_parts) < 2:
+ return {"outputText": f"Error: sourceRepo must be owner/repo format, got '{source_repo}'"}
+ repo_owner_name = f"{source_parts[0]}/{source_parts[1]}"
+ repo_short_name = source_parts[1]
+ source_path_prefix = "/".join(source_parts[2:]) if len(source_parts) > 2 else ""
+
+ code_namespace = f"files:{repo_owner_name}"
+
+ # Reconstruct the download input: owner/repo/path (short form for the download agent)
+ download_source = repo_owner_name
+ if source_path_prefix:
+ download_source += f"/{source_path_prefix}"
+
+ # Build namespace list: code namespace + any supplemental
+ namespaces = [code_namespace]
+ if supplemental_data:
+ for ns in supplemental_data.split(","):
+ ns = ns.strip()
+ if ns and ns not in namespaces:
+ namespaces.append(ns)
+
+ # Fetch latest commit hash and append repo/hash to output directory
+ source_headers = {"Accept": "application/vnd.github+json", "X-GitHub-Api-Version": "2022-11-28"}
+ if source_token:
+ source_headers["Authorization"] = f"Bearer {source_token}"
+ try:
+ commits_resp = await http_client.get(
+ f"https://api.github.com/repos/{repo_owner_name}/commits?per_page=1",
+ headers=source_headers,
+ )
+ commits_data = commits_resp.json()
+ commit_hash = commits_data[0]["sha"][:7]
+ except Exception as e:
+ print(f" WARNING: Could not fetch commit hash ({e}), using 'latest'", flush=True)
+ commit_hash = "latest"
+
+ output_directory = f"{output_directory.strip('/')}/{repo_short_name}/{commit_hash}"
+ print(f" Output directory: {output_directory}", flush=True)
+
+ all_outputs = []
+ successes = []
+ failures = []
+ report_directories = []
+
+ # =============================================================
+ # Helper: filter ASVS sections by level
+ # =============================================================
+ asvs_level_cache = {}
+
+ def load_asvs_levels():
+ if asvs_level_cache:
+ return
+ try:
+ asvs_ns = data_store.use_namespace("asvs")
+ all_keys = asvs_ns.list_keys()
+ req_keys = [k for k in all_keys if k.startswith("asvs:requirements:")]
+ for rk in req_keys:
+ req = asvs_ns.get(rk)
+ if req:
+ section_id = rk.replace("asvs:requirements:", "")
+ asvs_level_cache[section_id] = int(req.get("level", 1))
+ print(f" Loaded ASVS levels for {len(asvs_level_cache)} sections", flush=True)
+ except Exception as e:
+ print(f" WARNING: Could not load ASVS levels: {e}", flush=True)
+
+ def filter_sections_by_level(sections):
+ if not level:
+ return sections
+ load_asvs_levels()
+ return [s for s in sections if asvs_level_cache.get(s, 1) <= max_level_num]
+
+ # =============================================================
+ # Step 1: Download source code
+ # =============================================================
+ print(f"{'='*60}", flush=True)
+ print(f"Step 1: Downloading source code", flush=True)
+ print(f" Source: {source_repo}", flush=True)
+ print(f" Repo: {repo_owner_name}", flush=True)
+ if source_path_prefix:
+ print(f" Path: {source_path_prefix}", flush=True)
+ print(f" Namespace: {code_namespace}", flush=True)
+ print(f"{'='*60}", flush=True)
+
+ download_input = download_source
+ if source_token:
+ download_input += f"\n{source_token}"
+
+ try:
+ download_result = await gofannon_client.call(
+ agent_name="download_github_repo_to_datastore",
+ input_dict={
+ "inputText": download_input,
+ }
+ )
+ download_output = download_result.get("outputText", "")
+ print(f" {download_output}", flush=True)
+ except Exception as e:
+ return {"outputText": f"Download failed: {e}"}
+
+ # =============================================================
+ # Step 2: Discover architecture
+ # =============================================================
+ if discover:
+ print(f"\n{'='*60}", flush=True)
+ print(f"Step 2: Discovering codebase architecture", flush=True)
+ print(f" Namespaces: {namespaces}", flush=True)
+ print(f"{'='*60}", flush=True)
+
+ try:
+ discovery_result = await gofannon_client.call(
+ agent_name="discover_codebase_architecture",
+ input_dict={
+ "inputNamespace": ",".join(namespaces),
+ }
+ )
+ pass_config = json.loads(discovery_result.get("outputText", "{}"))
+ if "error" in pass_config:
+ return {"outputText": f"Discovery failed: {pass_config['error']}"}
+
+ passes = pass_config.get("passes", [])
+ false_positive_guidance = pass_config.get("false_positive_guidance", [])
+ domain_groups = pass_config.get("domain_groups", {})
+ print(f" Discovery complete: {len(passes)} passes", flush=True)
+ except Exception as e:
+ return {"outputText": f"Discovery agent failed: {e}"}
+
+ # Filter by level
+ for pass_def in passes:
+ pass_def["asvs_sections"] = filter_sections_by_level(pass_def.get("asvs_sections", []))
+ passes = [p for p in passes if p.get("asvs_sections")]
+
+ if level:
+ for dn in list(domain_groups.keys()):
+ domain_groups[dn] = filter_sections_by_level(domain_groups[dn])
+ domain_groups = {k: v for k, v in domain_groups.items() if v}
+
+ total_sections = sum(len(p.get("asvs_sections", [])) for p in passes)
+ print(f" After level filter ({level or 'all'}): {total_sections} sections, {len(passes)} passes", flush=True)
+
+ if total_sections == 0:
+ return {"outputText": f"No ASVS sections match level {level}."}
+
+ # =============================================================
+ # Step 3: Audit + push
+ # =============================================================
+ print(f"\n{'='*60}", flush=True)
+ print(f"Step 3: Auditing {total_sections} sections", flush=True)
+ print(f"{'='*60}", flush=True)
+
+ section_idx = 0
+ for pass_def in passes:
+ pass_name = pass_def.get("name", "unknown")
+ sections = pass_def.get("asvs_sections", [])
+ include_files = pass_def.get("files", [])
+ domain_context = pass_def.get("domain_context", "")
+ pass_output_dir = f"{output_directory}/{pass_name}" if output_directory else pass_name
+ report_directories.append(pass_output_dir)
+
+ print(f"\n{'='*60}", flush=True)
+ print(f"Pass: {pass_name} ({len(sections)} sections)", flush=True)
+ print(f"{'='*60}", flush=True)
+
+ for section in sections:
+ section_idx += 1
+ print(f"\n[{section_idx}/{total_sections}] {section} ({pass_name})", flush=True)
+
+ audit_output_text = None
+ try:
+ audit_result = await gofannon_client.call(
+ agent_name="run_asvs_security_audit",
+ input_dict={
+ "inputText": json.dumps({
+ "namespaces": namespaces,
+ "asvs": section,
+ "includeFiles": include_files,
+ "domainContext": domain_context,
+ "severityThreshold": severity_threshold,
+ "falsePositiveGuidance": false_positive_guidance,
+ })
+ }
+ )
+ audit_output_text = audit_result.get("outputText", "")
+ print(f" Audit done: {len(audit_output_text)} chars", flush=True)
+ except Exception as e:
+ print(f" Audit FAILED: {e}", flush=True)
+ failures.append(f"{section} (audit): {e}")
+ continue
+
+ try:
+ await gofannon_client.call(
+ agent_name="add_markdown_file_to_github_directory",
+ input_dict={
+ "inputText": json.dumps({
+ "repo": output_repo,
+ "token": output_token,
+ "directory": pass_output_dir,
+ "filename": f"{section}.md",
+ }),
+ "commitMessage": f"ASVS {level or 'full'} audit: {section} ({pass_name})",
+ "fileContents": audit_output_text,
+ }
+ )
+ print(f" Push OK", flush=True)
+ successes.append(section)
+ except Exception as e:
+ print(f" Push FAILED: {e}", flush=True)
+ failures.append(f"{section} (push): {e}")
+ all_outputs.append(audit_output_text)
+
+ # =============================================================
+ # Step 4: Consolidate
+ # =============================================================
+ if consolidate and successes:
+ print(f"\n{'='*60}", flush=True)
+ print(f"Step 4: Consolidating reports", flush=True)
+ print(f"{'='*60}", flush=True)
+ try:
+ await gofannon_client.call(
+ agent_name="consolidate_asvs_security_audit_reports",
+ input_dict={
+ "inputText": "\n".join([
+ f"repo: {output_repo}",
+ f"pat: {output_token}",
+ f"directories: {', '.join(report_directories)}",
+ f"output: {output_directory}",
+ ]),
+ "domainGroups": json.dumps(domain_groups),
+ "level": level or "L3",
+ }
+ )
+ print(f" Consolidation done", flush=True)
+ except Exception as e:
+ print(f" Consolidation FAILED: {e}", flush=True)
+ failures.append(f"consolidation: {e}")
+ elif not successes:
+ print(f"\n Skipping consolidation — no successful audits", flush=True)
+
+ else:
+ # =============================================================
+ # NO-DISCOVER MODE: just audit all ASVS sections in the data
+ # store without domain scoping. Useful for small codebases.
+ # =============================================================
+ print(f"\n{'='*60}", flush=True)
+ print(f"Step 2: Loading ASVS sections (no discovery)", flush=True)
+ print(f"{'='*60}", flush=True)
+
+ # Get all ASVS sections from the data store, filtered by level
+ all_sections = []
+ try:
+ asvs_ns = data_store.use_namespace("asvs")
+ all_keys = asvs_ns.list_keys()
+ req_keys = [k for k in all_keys if k.startswith("asvs:requirements:")]
+ for rk in sorted(req_keys):
+ section_id = rk.replace("asvs:requirements:", "")
+ all_sections.append(section_id)
+ except Exception as e:
+ return {"outputText": f"Could not load ASVS sections: {e}"}
+
+ sections = filter_sections_by_level(all_sections)
+ print(f" {len(sections)} sections to audit (level: {level or 'all'})", flush=True)
+
+ for i, section in enumerate(sections):
+ print(f"\n[{i+1}/{len(sections)}] Section {section}", flush=True)
+ audit_output_text = None
+ try:
+ audit_result = await gofannon_client.call(
+ agent_name="run_asvs_security_audit",
+ input_dict={
+ "inputText": json.dumps({
+ "namespaces": namespaces,
+ "asvs": section,
+ })
+ }
+ )
+ audit_output_text = audit_result.get("outputText", "")
+ except Exception as e:
+ failures.append(f"{section} (audit): {e}")
+ continue
+ try:
+ await gofannon_client.call(
+ agent_name="add_markdown_file_to_github_directory",
+ input_dict={
+ "inputText": json.dumps({
+ "repo": output_repo,
+ "token": output_token,
+ "directory": output_directory,
+ "filename": f"{section}.md",
+ }),
+ "commitMessage": f"ASVS audit: {section}",
+ "fileContents": audit_output_text,
+ }
+ )
+ successes.append(section)
+ except Exception as e:
+ failures.append(f"{section} (push): {e}")
+ all_outputs.append(audit_output_text)
+
+ # =============================================================
+ # Summary
+ # =============================================================
+ print(f"\n{'='*60}", flush=True)
+ print(f"Complete: {len(successes)} succeeded, {len(failures)} failed", flush=True)
+ if failures:
+ for f in failures:
+ print(f" - {f}", flush=True)
+ return {"outputText": "\n\n---\n\n".join(all_outputs)}
+ finally:
+ await http_client.aclose()
\ No newline at end of file
diff --git a/repos/apache/ASVS/agents/run_asvs_security_audit.py b/repos/apache/ASVS/agents/run_asvs_security_audit.py
new file mode 100644
index 0000000..2e35855
--- /dev/null
+++ b/repos/apache/ASVS/agents/run_asvs_security_audit.py
@@ -0,0 +1,947 @@
+# run_asvs_security_audit
+from agent_factory.remote_mcp_client import RemoteMCPClient
+from services.llm_service import call_llm
+import httpx
+
+async def run(input_dict, tools):
+ mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+ http_client = httpx.AsyncClient()
+ try:
+ import json
+ import re
+ import fnmatch
+ from datetime import date
+ audit_date = date.today().strftime("%b %d, %Y")
+
+ # Parse input
+ input_text = input_dict.get("inputText", "")
+ params = {}
+ if input_text.strip().startswith('{'):
+ try:
+ params = json.loads(input_text)
+ except json.JSONDecodeError:
+ pass
+
+ if not params.get('namespace') and not params.get('namespaces'):
+ match = re.search(r'(?:namespaces?|ns)[:\s]+([^\n]+?)(?:\s+asvs|$)', input_text, re.IGNORECASE)
+ if match:
+ raw = match.group(1).strip()
+ if ',' in raw:
+ params['namespaces'] = [n.strip() for n in raw.split(',')]
+ else:
+ params['namespace'] = raw
+ else:
+ match = re.search(r'(?:namespace|ns)[:\s]+([^\s,]+)', input_text, re.IGNORECASE)
+ if match:
+ params['namespace'] = match.group(1)
+
+ if not params.get('asvs') and not params.get('asvs_section'):
+ match = re.search(r'(?:asvs[-_]?section|asvs)[:\s]+(\d+(?:\.\d+)*)', input_text, re.IGNORECASE)
+ if match:
+ params['asvs'] = match.group(1)
+
+ namespaces = params.get('namespaces') or ([params.get('namespace')] if params.get('namespace') else [])
+ asvs = params.get('asvs') or params.get('asvs_section', '')
+
+ # --- NEW INPUTS ---
+ include_files = params.get('includeFiles', [])
+ severity_threshold = params.get('severityThreshold', '')
+ domain_context = params.get('domainContext', '')
+ false_positive_guidance = params.get('falsePositiveGuidance', [])
+
+ if not namespaces:
+ all_ns = data_store.list_namespaces()
+ file_ns = [ns for ns in all_ns if ns.startswith("files:")]
+ if file_ns:
+ namespaces = file_ns
+
+ if not namespaces:
+ return {"outputText": f"Error: No namespaces provided or found in data store. Available namespaces: {data_store.list_namespaces()}"}
+
+ if not asvs:
+ return {"outputText": "Error: No ASVS requirement specified. Please provide an asvs parameter (e.g., asvs: 5.3.2)"}
+
+ repo_name = "unknown"
+ for ns in namespaces:
+ if ns.startswith("files:"):
+ repo_name = ns.replace("files:", "")
+ break
+
+ print(f"Namespaces: {namespaces}", flush=True)
+ print(f"ASVS: {asvs}", flush=True)
+ if include_files:
+ print(f"File scope: {len(include_files)} patterns", flush=True)
+ if severity_threshold:
+ print(f"Severity threshold: {severity_threshold}", flush=True)
+
+ # =============================================================
+ # Model configuration
+ # =============================================================
+ SONNET_PROVIDER = "bedrock"
+ SONNET_MODEL = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
+ SONNET_PARAMS = {"temperature": 0.7, "max_tokens": 16384}
+
+ OPUS_PROVIDER = "bedrock"
+ OPUS_MODEL = "us.anthropic.claude-opus-4-6-v1"
+ OPUS_PARAMS = {"temperature": 1, "reasoning_effort": "medium", "max_tokens": 64000}
+
+ SONNET_CONTEXT = get_context_window(SONNET_PROVIDER, SONNET_MODEL)
+ OPUS_CONTEXT = get_context_window(OPUS_PROVIDER, OPUS_MODEL)
+
+ sonnet_semaphore = asyncio.Semaphore(5)
+ opus_semaphore = asyncio.Semaphore(2)
+
+ cache_key_prefix = f"asvs-{asvs}-{'-'.join(namespaces)}"
+ relevance_cache_ns = data_store.use_namespace(f"audit-cache:relevance:{cache_key_prefix}")
+ inventory_cache_ns = data_store.use_namespace(f"audit-cache:inventory:{cache_key_prefix}")
+ analysis_cache_ns = data_store.use_namespace(f"audit-cache:analysis:{cache_key_prefix}")
+
+ # =============================================================
+ # Step 0: Load ASVS requirement context from data store
+ # =============================================================
+ print("\n=== Step 0: Loading ASVS requirement context ===", flush=True)
+ asvs_description = ""
+ try:
+ asvs_ns = data_store.use_namespace("asvs")
+ req = asvs_ns.get(f"asvs:requirements:{asvs}")
+ if req:
+ parts = [f"ASVS Requirement {asvs}"]
+ if req.get("req_description"):
+ parts.append(f"Description: {req['req_description']}")
+ if req.get("level"):
+ parts.append(f"Level: {req['level']}")
+ sec_id = req.get("section_id", "")
+ if sec_id:
+ sec = asvs_ns.get(f"asvs:sections:{sec_id}")
+ if sec:
+ parts.append(f"Section: {sec.get('section_name', '')}")
+ if sec.get("description"):
+ parts.append(f"Section Description: {sec['description']}")
+ ch_id = req.get("chapter_id", "")
+ if ch_id:
+ ch = asvs_ns.get(f"asvs:chapters:{ch_id}")
+ if ch:
+ parts.append(f"Chapter: {ch.get('chapter_name', '')}")
+ if ch.get("control_objective"):
+ parts.append(f"Control Objective: {ch['control_objective']}")
+ asvs_description = "\n".join(parts)
+ print(f" Loaded: {asvs_description[:200]}", flush=True)
+ else:
+ print(f" WARNING: No data found for asvs:requirements:{asvs}", flush=True)
+ except Exception as e:
+ print(f" ERROR reading ASVS data store: {e}", flush=True)
+
+ if not asvs_description:
+ asvs_description = f"ASVS Requirement {asvs} (requirement description not available in data store)"
+
+ # =============================================================
+ # Step 1: Read files from data store (with optional pre-scoping)
+ # =============================================================
+ print("\n=== Step 1: Reading files from data store ===", flush=True)
+
+ all_files = {}
+ for ns in namespaces:
+ ns_store = data_store.use_namespace(ns)
+ keys = ns_store.list_keys()
+
+ # --- NEW: Pre-scope to includeFiles if provided ---
+ if include_files:
+ keys = [k for k in keys if any(
+ fnmatch.fnmatch(k, pattern) for pattern in include_files
+ )]
+ print(f"Namespace '{ns}': {len(keys)} keys (scoped by includeFiles)", flush=True)
+ else:
+ print(f"Namespace '{ns}': {len(keys)} keys", flush=True)
+
+ file_contents = ns_store.get_many(keys) if keys else {}
+ for k, v in file_contents.items():
+ if v is not None:
+ content = v if isinstance(v, str) else json.dumps(v, default=str) if v else ""
+ all_files[k] = content
+
+ print(f"Total files loaded: {len(all_files)}", flush=True)
+
+ if not all_files:
+ return {"outputText": f"Error: No files found in namespaces {namespaces}. Available namespaces: {data_store.list_namespaces()}"}
+
+ # Skip non-code files
+ SKIP_DIRS = {
+ 'node_modules', 'vendor', 'third_party', 'third-party',
+ 'dist', 'build', 'out', 'target',
+ '__pycache__', '.pytest_cache', '.mypy_cache', 'coverage', '.next', '.nuxt',
+ 'assets', 'images', 'img', 'static/images', 'static/fonts', 'static/webfonts',
+ 'public/images', 'fonts', 'webfonts',
+ '.github/workflows',
+ 'venv', '.venv', 'env', '.env',
+ '.git', '.idea', '.vscode',
+ }
+ SKIP_FILES = {
+ 'package-lock.json', 'yarn.lock', 'poetry.lock', 'Cargo.lock',
+ 'composer.lock', 'pnpm-lock.yaml', 'Gemfile.lock', 'uv.lock',
+ 'LICENSE', 'LICENSE.md', 'LICENSE.txt',
+ 'README.md', 'README.rst', 'README.txt', 'README',
+ 'CHANGELOG.md', 'CHANGELOG', 'CONTRIBUTING.md', 'CODE_OF_CONDUCT.md',
+ '.gitignore', '.dockerignore', '.prettierrc', '.eslintrc', '.editorconfig',
+ '.npmrc', '.yarnrc',
+ }
+ SKIP_EXTENSIONS = {
+ '.min.js', '.min.css', '.bundle.js', '.bundle.css',
+ '.map', '.css', '.scss', '.less', '.sass', '.styl',
+ '.woff', '.woff2', '.ttf', '.eot', '.otf',
+ '.png', '.jpg', '.jpeg', '.gif', '.svg', '.ico', '.webp', '.bmp',
+ '.mp3', '.mp4', '.wav', '.avi', '.mov', '.webm', '.ogg',
+ '.pdf', '.doc', '.docx', '.xls', '.xlsx', '.ppt', '.pptx',
+ '.zip', '.tar', '.gz', '.rar', '.7z', '.lock',
+ '.exe', '.dll', '.so', '.dylib', '.pyc', '.pyo', '.class', '.o', '.obj',
+ }
+
+ def should_skip_file(filepath):
+ path_lower = filepath.lower()
+ parts = filepath.split('/')
+ for part in parts[:-1]:
+ if part.lower() in SKIP_DIRS:
+ return True
+ filename = parts[-1] if parts else ''
+ if filename in SKIP_FILES:
+ return True
+ for ext in SKIP_EXTENSIONS:
+ if path_lower.endswith(ext):
+ return True
+ return False
+
+ filtered_files = {}
+ skipped_count = 0
+ for key, content in all_files.items():
+ if should_skip_file(key):
+ skipped_count += 1
+ continue
+ filtered_files[key] = content
+
+ print(f"Filtered: {len(filtered_files)} files to analyze, {skipped_count} skipped", flush=True)
+
+ if not filtered_files:
+ return {"outputText": f"Error: All {len(all_files)} files were filtered out. Check skip rules."}
+
+ # =============================================================
+ # Step 2: Relevance filtering (SKIPPED when includeFiles provided)
+ # =============================================================
+ if include_files:
+ print("\n=== Step 2: Relevance filtering SKIPPED (includeFiles provided) ===", flush=True)
+ relevant_files = filtered_files
+ relevance_scores = {path: 10 for path in filtered_files}
+ sorted_relevant = sorted(relevant_files.keys())
+ else:
+ print("\n=== Step 2: Relevance filtering (Sonnet, parallel) ===", flush=True)
+
+ cached_relevance = relevance_cache_ns.get("scores")
+ if cached_relevance:
+ relevance_scores = cached_relevance
+ print(f" Using cached relevance scores for {len(relevance_scores)} files", flush=True)
+ else:
+ file_previews = {}
+ for path, content in filtered_files.items():
+ lines = content.split('\n')
+ file_previews[path] = '\n'.join(lines[:200])
+
+ SAFE_SONNET_LIMIT = int(SONNET_CONTEXT * 0.40)
+
+ relevance_prompt_template = f"""You are a security auditor performing file relevance filtering.
+
+ASVS Requirement: {asvs}
+{asvs_description}
+
+Below are file paths with previews (first ~200 lines) from a codebase.
+Rate each file's relevance to the ASVS requirement above on a scale of 0-10:
+- 10: Directly implements or should implement controls for this requirement
+- 7-9: Contains related security controls, data handling, or configuration
+- 4-6: May contain relevant patterns indirectly
+- 1-3: Unlikely to be relevant
+- 0: Definitely not relevant
+
+Return ONLY a JSON object mapping file paths to relevance scores (integer 0-10).
+Example: {{"src/auth.py": 9, "src/utils.py": 3}}
+
+FILES TO EVALUATE:
+"""
+
+ template_tokens = count_tokens(relevance_prompt_template, SONNET_PROVIDER, SONNET_MODEL)
+ preview_budget = SAFE_SONNET_LIMIT - template_tokens
+
+ preview_batches = []
+ current_batch = {}
+ current_tokens = 0
+ for path, preview in file_previews.items():
+ entry = f"\n--- {path} ---\n{preview}\n"
+ entry_tokens = count_tokens(entry, SONNET_PROVIDER, SONNET_MODEL)
+ if current_tokens + entry_tokens > preview_budget and current_batch:
+ preview_batches.append(current_batch)
+ current_batch = {}
+ current_tokens = 0
+ current_batch[path] = entry
+ current_tokens += entry_tokens
+ if current_batch:
+ preview_batches.append(current_batch)
+
+ print(f" Relevance filtering: {len(preview_batches)} batches", flush=True)
+
+ relevance_scores = {}
+
+ async def filter_batch(i, batch):
+ async with sonnet_semaphore:
+ entries_text = "".join(batch.values())
+ prompt = relevance_prompt_template + entries_text
+ messages = [{"role": "user", "content": prompt}]
+ for attempt in range(2):
+ try:
+ content_resp, _ = await call_llm(
+ provider=SONNET_PROVIDER, model=SONNET_MODEL,
+ messages=messages, parameters=SONNET_PARAMS,
+ timeout=120,
+ )
+ json_match = re.search(r'\{[^{}]*(?:\{[^{}]*\}[^{}]*)*\}', content_resp, re.DOTALL)
+ if json_match:
+ scores = json.loads(json_match.group())
+ print(f" Batch {i+1}: scored {len(scores)} files", flush=True)
+ return scores
+ except Exception as e:
+ if attempt == 0:
+ print(f" Batch {i+1} attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+ await asyncio.sleep(5)
+ else:
+ print(f" Batch {i+1} FAILED: {e}", flush=True)
+ print(f" WARNING: Batch {i+1} defaulting {len(batch)} files to score=5", flush=True)
+ return {path: 5 for path in batch}
+
+ batch_results = await asyncio.gather(*[
+ filter_batch(i, batch)
+ for i, batch in enumerate(preview_batches)
+ ])
+ for scores in batch_results:
+ relevance_scores.update(scores)
+
+ relevance_cache_ns.set("scores", relevance_scores)
+
+ relevant_files = {}
+ for path, content in filtered_files.items():
+ score = relevance_scores.get(path, 5)
+ if isinstance(score, (int, float)) and score >= 4:
+ relevant_files[path] = content
+
+ if len(relevant_files) < 3 and filtered_files:
+ for path, content in filtered_files.items():
+ score = relevance_scores.get(path, 5)
+ if isinstance(score, (int, float)) and score >= 2:
+ relevant_files[path] = content
+
+ sorted_relevant = sorted(relevant_files.keys(), key=lambda p: relevance_scores.get(p, 0), reverse=True)
+
+ print(f" Relevant files: {len(relevant_files)} (from {len(filtered_files)})", flush=True)
+
+ # =============================================================
+ # Step 3: Code inventory (Sonnet, parallel, checkpointed)
+ # =============================================================
+ print("\n=== Step 3: Code inventory (Sonnet, parallel) ===", flush=True)
+
+ cached_inventory = inventory_cache_ns.get("result")
+ if cached_inventory:
+ code_inventory = cached_inventory
+ print(f" Using cached inventory ({len(code_inventory)} chars)", flush=True)
+ else:
+ inventory_prompt_template = f"""You are a security code analyst. Extract a structured code inventory from each file below.
+
+For each file, produce:
+1. **Imports** — list all imports (especially security-related: auth, crypto, validators, sanitizers, path handling)
+2. **Classes** — name, base classes, key methods with signatures
+3. **Functions** — name, parameters (with types if available), decorators, line numbers (approximate)
+4. **Security-relevant patterns** — validators, auth checks, path operations, file operations, crypto operations
+5. **Routes/endpoints** — any URL routing, decorators like @app.route, @router.get, etc.
+6. **Configuration** — security-relevant config values, environment variables
+
+Return the inventory as structured markdown. Be thorough but concise.
+
+FILES:
+"""
+
+ SAFE_SONNET_LIMIT_INV = int(SONNET_CONTEXT * 0.40)
+ inv_template_tokens = count_tokens(inventory_prompt_template, SONNET_PROVIDER, SONNET_MODEL)
+ inv_budget = SAFE_SONNET_LIMIT_INV - inv_template_tokens
+
+ inv_batches = []
+ current_batch = {}
+ current_tokens = 0
+ for path in sorted_relevant:
+ content = relevant_files[path]
+ entry = f"\n--- {path} ---\n{content}\n"
+ entry_tokens = count_tokens(entry, SONNET_PROVIDER, SONNET_MODEL)
+ if entry_tokens > inv_budget:
+ if current_batch:
+ inv_batches.append(current_batch)
+ current_batch = {}
+ current_tokens = 0
+ inv_batches.append({path: entry})
+ continue
+ if current_tokens + entry_tokens > inv_budget and current_batch:
+ inv_batches.append(current_batch)
+ current_batch = {}
+ current_tokens = 0
+ current_batch[path] = entry
+ current_tokens += entry_tokens
+ if current_batch:
+ inv_batches.append(current_batch)
+
+ print(f" Code inventory: {len(inv_batches)} batches", flush=True)
+
+ async def inventory_batch(i, batch):
+ async with sonnet_semaphore:
+ entries_text = "".join(batch.values())
+ prompt = inventory_prompt_template + entries_text
+ messages = [{"role": "user", "content": prompt}]
+
+ msg_tokens = count_message_tokens(messages, SONNET_PROVIDER, SONNET_MODEL)
+ if msg_tokens > int(SONNET_CONTEXT * 0.80):
+ items = list(batch.items())
+ mid = len(items) // 2
+ results = []
+ for sub_items in [items[:mid], items[mid:]]:
+ sub_text = "".join([v for _, v in sub_items])
+ sub_messages = [{"role": "user", "content": inventory_prompt_template + sub_text}]
+ try:
+ resp, _ = await call_llm(
+ provider=SONNET_PROVIDER, model=SONNET_MODEL,
+ messages=sub_messages, parameters=SONNET_PARAMS,
+ timeout=300,
+ )
+ results.append(resp)
+ except Exception as e:
+ print(f" Inventory sub-batch failed: {e}", flush=True)
+ return "\n\n".join(results)
+
+ for attempt in range(2):
+ try:
+ resp, _ = await call_llm(
+ provider=SONNET_PROVIDER, model=SONNET_MODEL,
+ messages=messages, parameters=SONNET_PARAMS,
+ timeout=300,
+ )
+ print(f" Inventory batch {i+1} complete", flush=True)
+ return resp
+ except Exception as e:
+ if attempt == 0:
+ print(f" Inventory batch {i+1} attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+ await asyncio.sleep(5)
+ else:
+ print(f" Inventory batch {i+1} FAILED: {e}", flush=True)
+ return ""
+
+ inventory_results = await asyncio.gather(*[
+ inventory_batch(i, batch)
+ for i, batch in enumerate(inv_batches)
+ ])
+ code_inventory = "\n\n---\n\n".join([r for r in inventory_results if r])
+ inventory_cache_ns.set("result", code_inventory)
+
+ print(f" Inventory total: {len(code_inventory)} chars", flush=True)
+
+ # =============================================================
+ # Step 4: Deep security analysis (Opus, checkpointed, with retry)
+ # =============================================================
+ print("\n=== Step 4: Deep security analysis (Opus) ===", flush=True)
+
+ # --- BUILD SYSTEM PROMPT WITH NEW INPUTS ---
+ analysis_system_prompt = f"""You are an expert application security auditor performing a comprehensive security audit against ASVS requirements.
+
+## ASVS Requirement Under Audit
+{asvs_description}
+"""
+
+ # Inject domain context if provided
+ if domain_context:
+ analysis_system_prompt += f"""
+## Domain Context
+{domain_context}
+"""
+
+ # Inject severity threshold if provided
+ if severity_threshold:
+ severity_levels = {"CRITICAL": 4, "HIGH": 3, "MEDIUM": 2, "LOW": 1}
+ threshold_val = severity_levels.get(severity_threshold.upper(), 0)
+ if threshold_val > 0:
+ included = [k for k, v in severity_levels.items() if v >= threshold_val]
+ analysis_system_prompt += f"""
+## Severity Threshold
+Only report findings at these severity levels: {', '.join(included)}.
+Do not include findings below {severity_threshold.upper()} severity.
+"""
+
+ # Inject false positive guidance if provided
+ if false_positive_guidance:
+ guidance_text = "\n".join(f"- {g}" for g in false_positive_guidance)
+ analysis_system_prompt += f"""
+## Known False Positive Patterns (DO NOT FLAG)
+The following patterns are intentional design decisions in this codebase. Do not report them as vulnerabilities:
+{guidance_text}
+"""
+
+ analysis_system_prompt += """
+## Audit Instructions
+
+Follow ALL of these analysis requirements:
+
+### Core Principle: Existence ≠ Application
+For each security control found:
+- Document where it's DEFINED
+- Map ALL entry points that should use it
+- Verify it's actually CALLED at each entry point
+- Flag coverage gaps (control exists but not applied = CRITICAL)
+
+### Gap Type Classification
+| Gap Type | Description | Severity |
+|----------|-------------|----------|
+| Type A | Entry point with NO control | Standard vulnerability |
+| Type B | Control EXISTS but NOT CALLED | CRITICAL (false confidence) |
+| Type C | Control CALLED but RESULT IGNORED | CRITICAL |
+| Type D | Control CALLED but AFTER sensitive operation | CRITICAL |
+
+### Related Function Analysis
+When you find a vulnerability, IMMEDIATELY search for:
+- Singular/plural variants
+- Sync/async variants
+- Public/private variants
+- Same-file functions with similar parameters or operations
+
+### False Positive Prevention
+Before finalizing each finding:
+1. Where does this input ACTUALLY originate? Is it truly user-controllable?
+2. Is there validation applied EARLIER in the call chain?
+3. Can an external attacker actually control this value?
+4. If you listed something as a positive pattern, don't also list it as a vulnerability
+
+### Exclusions
+Do NOT report:
+- Database-sourced values without injection path
+- Already-validated inputs
+- Developer tooling/test code
+- Issues requiring prior compromise
+- Theoretical issues without specific exploit
+- Test/example code
+
+### Output Requirements
+For each finding, provide:
+- Severity level (CRITICAL/HIGH/MEDIUM/LOW)
+- Finding ID (format: ASVS-{asvs.replace('.', '')}-SEV-NNN)
+- Exact file location and function name with line numbers
+- Vulnerable code quote
+- Data flow (source → sink → missing control)
+- Proof of concept (specific malicious request)
+- Impact description
+- Remediation with code example
+
+Also provide:
+- Security Controls Inventory with coverage analysis
+- Critical File Review tables
+- Positive Security Patterns
+- Architecture Observations
+- Prioritized Recommendations
+
+Be thorough but precise. If something is done correctly, acknowledge it as a positive pattern - don't invent issues."""
+
+ SAFE_OPUS_LIMIT = int(OPUS_CONTEXT * 0.40)
+ system_tokens = count_tokens(analysis_system_prompt, OPUS_PROVIDER, OPUS_MODEL)
+ inventory_section = f"\n\n## Code Inventory (extracted by pre-analysis)\n\n{code_inventory}\n\n"
+ inventory_tokens = count_tokens(inventory_section, OPUS_PROVIDER, OPUS_MODEL)
+ user_template = "## Source Code Files\n\nAnalyze the following files for security issues related to the ASVS requirement:\n\n"
+ user_template_tokens = count_tokens(user_template, OPUS_PROVIDER, OPUS_MODEL)
+
+ max_inv_tokens = int(SAFE_OPUS_LIMIT * 0.15)
+ if inventory_tokens > max_inv_tokens:
+ print(f" Truncating inventory from {inventory_tokens} to {max_inv_tokens} tokens", flush=True)
+ inv_lines = code_inventory.split('\n')
+ truncated_inv = ""
+ for line in inv_lines:
+ candidate = truncated_inv + line + "\n"
+ if count_tokens(candidate, OPUS_PROVIDER, OPUS_MODEL) > max_inv_tokens:
+ break
+ truncated_inv = candidate
+ inventory_section = f"\n\n## Code Inventory (extracted by pre-analysis, truncated)\n\n{truncated_inv}\n\n"
+ inventory_tokens = count_tokens(inventory_section, OPUS_PROVIDER, OPUS_MODEL)
+
+ opus_content_budget = SAFE_OPUS_LIMIT - system_tokens - inventory_tokens - user_template_tokens
+
+ print(f" Opus content budget: {opus_content_budget} tokens", flush=True)
+
+ opus_batches = []
+ current_batch = {}
+ current_tokens = 0
+ for path in sorted_relevant:
+ content = relevant_files[path]
+ entry = f"\n### File: `{path}`\n```\n{content}\n```\n"
+ entry_tokens = count_tokens(entry, OPUS_PROVIDER, OPUS_MODEL)
+ if entry_tokens > opus_content_budget:
+ if current_batch:
+ opus_batches.append(current_batch)
+ current_batch = {}
+ current_tokens = 0
+ opus_batches.append({path: entry})
+ continue
+ if current_tokens + entry_tokens > opus_content_budget and current_batch:
+ opus_batches.append(current_batch)
+ current_batch = {}
+ current_tokens = 0
+ current_batch[path] = entry
+ current_tokens += entry_tokens
+ if current_batch:
+ opus_batches.append(current_batch)
+
+ print(f" Deep analysis: {len(opus_batches)} batches", flush=True)
+
+ async def analyze_batch(i, batch):
+ cache_key = f"batch-{i}"
+ cached = analysis_cache_ns.get(cache_key)
+ if cached:
+ print(f" Opus batch {i+1}: cached", flush=True)
+ return cached
+
+ async with opus_semaphore:
+ entries_text = "".join(batch.values())
+ user_content = user_template + entries_text + inventory_section
+ messages = [
+ {"role": "user", "content": analysis_system_prompt + "\n\n" + user_content}
+ ]
+
+ msg_tokens = count_message_tokens(messages, OPUS_PROVIDER, OPUS_MODEL)
+ limit = int(OPUS_CONTEXT * 0.80)
+ print(f" Opus batch {i+1}/{len(opus_batches)}: {msg_tokens} tokens, {len(batch)} files", flush=True)
+
+ if msg_tokens > limit:
+ items = list(batch.items())
+ if len(items) > 1:
+ mid = len(items) // 2
+ results = []
+ for half_label, half_items in [("a", items[:mid]), ("b", items[mid:])]:
+ half_text = "".join([v for _, v in half_items])
+ half_user = user_template + half_text + inventory_section
+ half_messages = [{"role": "user", "content": analysis_system_prompt + "\n\n" + half_user}]
+ for attempt in range(3):
+ try:
+ resp, _ = await call_llm(
+ provider=OPUS_PROVIDER, model=OPUS_MODEL,
+ messages=half_messages, parameters=OPUS_PARAMS,
+ timeout=1800,
+ )
+ results.append(resp)
+ print(f" Sub-batch {half_label} complete", flush=True)
+ break
+ except Exception as e:
+ if attempt < 2:
+ wait = 15 * (attempt + 1)
+ print(f" Sub-batch {half_label} attempt {attempt+1} failed ({type(e).__name__}), retrying in {wait}s...", flush=True)
+ await asyncio.sleep(wait)
+ else:
+ print(f" Sub-batch {half_label} FAILED after 3 attempts: {e}", flush=True)
+ results.append(f"[Analysis failed for sub-batch {i+1}{half_label}: {str(e)[:200]}]")
+ combined = "\n\n---\n\n".join(results)
+ analysis_cache_ns.set(cache_key, combined)
+ return combined
+ else:
+ key, entry_val = items[0]
+ slim_messages = [{"role": "user", "content": analysis_system_prompt + "\n\n" + user_template + entry_val}]
+ for attempt in range(3):
+ try:
+ resp, _ = await call_llm(
+ provider=OPUS_PROVIDER, model=OPUS_MODEL,
+ messages=slim_messages, parameters=OPUS_PARAMS,
+ timeout=1800,
+ )
+ analysis_cache_ns.set(cache_key, resp)
+ return resp
+ except Exception as e:
+ if attempt < 2:
+ wait = 15 * (attempt + 1)
+ print(f" Single file attempt {attempt+1} failed ({type(e).__name__}), retrying in {wait}s...", flush=True)
+ await asyncio.sleep(wait)
+ else:
+ print(f" Single file FAILED after 3 attempts: {e}", flush=True)
+ return f"[Analysis failed for {key}: {str(e)[:200]}]"
+
+ for attempt in range(3):
+ try:
+ resp, _ = await call_llm(
+ provider=OPUS_PROVIDER, model=OPUS_MODEL,
+ messages=messages, parameters=OPUS_PARAMS,
+ timeout=1800,
+ )
+ analysis_cache_ns.set(cache_key, resp)
+ print(f" Opus batch {i+1} complete", flush=True)
+ return resp
+ except Exception as e:
+ if attempt < 2:
+ wait = 15 * (attempt + 1)
+ print(f" Opus batch {i+1} attempt {attempt+1} failed ({type(e).__name__}), retrying in {wait}s...", flush=True)
+ await asyncio.sleep(wait)
+ else:
+ print(f" Opus batch {i+1} FAILED after 3 attempts: {e}", flush=True)
+ return f"[Analysis failed for batch {i+1}: {str(e)[:200]}]"
+
+ analysis_results = await asyncio.gather(*[
+ analyze_batch(i, batch)
+ for i, batch in enumerate(opus_batches)
+ ])
+
+ analysis_results = [r for r in analysis_results if r and not r.startswith("[Analysis failed")]
+ print(f" Analysis complete: {len(analysis_results)} results", flush=True)
+
+ if not analysis_results:
+ return {"outputText": "Error: All analysis batches failed. No results to report."}
+
+ # =============================================================
+ # Step 5: Consolidation if needed (Sonnet)
+ # =============================================================
+ if len(analysis_results) > 1:
+ print(f"\n=== Step 5: Consolidating {len(analysis_results)} results (Sonnet) ===", flush=True)
+
+ CONSOLIDATION_TEMPLATE = f"""You are consolidating multiple security audit batch results into a single unified analysis.
+
+ASVS Requirement: {asvs}
+{asvs_description}
+
+## Consolidation Rules:
+1. DEDUPLICATE - Merge findings describing the same vulnerability
+2. CHECK CONTRADICTIONS - If something appears as a positive pattern in ANY batch, remove from findings
+3. VERIFY DATA ORIGINS - Remove findings where source is database/config without user injection path
+4. CONSISTENT SEVERITY - Ensure similar issues have same severity
+5. REMOVE OUT-OF-SCOPE - Exclude test files, dev scripts
+6. VERIFY COMPLETENESS - For each vulnerability, confirm related functions were checked
+
+Consolidate these analysis results into a single comprehensive security audit report.
+Preserve all specific findings with their exact code references, but merge duplicates.
+Include a "Consolidation Notes" section documenting what was removed/merged and why.
+
+BATCH RESULTS TO CONSOLIDATE:
+"""
+
+ consolidation_params = {**SONNET_PARAMS, "max_tokens": 16384}
+ template_tokens_cons = count_tokens(CONSOLIDATION_TEMPLATE, SONNET_PROVIDER, SONNET_MODEL)
+ max_cons_content = int(SONNET_CONTEXT * 0.40) - template_tokens_cons
+
+ consolidation_round = 0
+ MAX_CONSOLIDATION_ROUNDS = 5
+ batch_results = analysis_results[:]
+
+ while len(batch_results) > 1:
+ consolidation_round += 1
+ prev_count = len(batch_results)
+ next_level = []
+ group = []
+ group_tokens = 0
+
+ for result in batch_results:
+ result_tokens = count_tokens(result, SONNET_PROVIDER, SONNET_MODEL)
+ if group and (group_tokens + result_tokens) > max_cons_content:
+ for attempt in range(2):
+ try:
+ cons_prompt = CONSOLIDATION_TEMPLATE + "\n---\n".join(group)
+ consolidated, _ = await call_llm(
+ provider=SONNET_PROVIDER, model=SONNET_MODEL,
+ messages=[{"role": "user", "content": cons_prompt}],
+ parameters=consolidation_params,
+ timeout=300,
+ )
+ next_level.append(consolidated)
+ break
+ except Exception as e:
+ if attempt == 0:
+ print(f" Consolidation attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+ await asyncio.sleep(5)
+ else:
+ print(f" Consolidation failed, keeping individual results", flush=True)
+ next_level.extend(group)
+ group = []
+ group_tokens = 0
+ group.append(result)
+ group_tokens += result_tokens
+
+ if group:
+ if len(group) == 1 and not next_level:
+ next_level.append(group[0])
+ else:
+ for attempt in range(2):
+ try:
+ cons_prompt = CONSOLIDATION_TEMPLATE + "\n---\n".join(group)
+ consolidated, _ = await call_llm(
+ provider=SONNET_PROVIDER, model=SONNET_MODEL,
+ messages=[{"role": "user", "content": cons_prompt}],
+ parameters=consolidation_params,
+ timeout=300,
+ )
+ next_level.append(consolidated)
+ break
+ except Exception as e:
+ if attempt == 0:
+ print(f" Consolidation attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+ await asyncio.sleep(5)
+ else:
+ print(f" Consolidation failed, keeping individual results", flush=True)
+ next_level.extend(group)
+
+ print(f" Round {consolidation_round}: {prev_count} -> {len(next_level)}", flush=True)
+ batch_results = next_level
+
+ if len(batch_results) >= prev_count:
+ print(f" No progress, stopping", flush=True)
+ break
+ if consolidation_round >= MAX_CONSOLIDATION_ROUNDS:
+ print(f" Max rounds reached", flush=True)
+ break
+
+ consolidated_analysis = "\n\n---\n\n".join(batch_results)
+ else:
+ consolidated_analysis = analysis_results[0]
+
+ # =============================================================
+ # Step 6: Format report (Sonnet)
+ # =============================================================
+ print("\n=== Step 6: Formatting report (Sonnet) ===", flush=True)
+
+ def count_findings(content):
+ counts = {"Critical": 0, "High": 0, "Medium": 0, "Low": 0}
+ counts["Critical"] = len(re.findall(r'###\s*\[CRITICAL\]', content, re.IGNORECASE))
+ counts["High"] = len(re.findall(r'###\s*\[HIGH\]', content, re.IGNORECASE))
+ counts["Medium"] = len(re.findall(r'###\s*\[MEDIUM\]', content, re.IGNORECASE))
+ counts["Low"] = len(re.findall(r'###\s*\[LOW\]', content, re.IGNORECASE))
+ if sum(counts.values()) == 0:
+ counts["Critical"] += len(re.findall(r'###[^\n]*-CRIT-', content, re.IGNORECASE))
+ counts["High"] += len(re.findall(r'###[^\n]*-HIGH-', content, re.IGNORECASE))
+ counts["Medium"] += len(re.findall(r'###[^\n]*-MED-', content, re.IGNORECASE))
+ counts["Low"] += len(re.findall(r'###[^\n]*-LOW-', content, re.IGNORECASE))
+ if sum(counts.values()) == 0:
+ counts["Critical"] += len(re.findall(r'\*\*Severity\*\*:\s*Critical', content, re.IGNORECASE))
+ counts["High"] += len(re.findall(r'\*\*Severity\*\*:\s*High', content, re.IGNORECASE))
+ counts["Medium"] += len(re.findall(r'\*\*Severity\*\*:\s*Medium', content, re.IGNORECASE))
+ counts["Low"] += len(re.findall(r'\*\*Severity\*\*:\s*Low', content, re.IGNORECASE))
+ return counts
+
+ findings_count = count_findings(consolidated_analysis)
+ file_list = "\n".join([f"- `{p}` (relevance: {relevance_scores.get(p, '?')})" for p in sorted_relevant])
+
+ format_prompt = f"""You are formatting a security audit report into clean, professional markdown.
+
+ASVS Requirement: {asvs}
+{asvs_description}
+
+## Formatting Requirements
+
+Format the analysis below into the following structure. Preserve ALL specific findings, code references,
+line numbers, and technical details exactly as provided. Do not add, remove, or modify any findings.
+
+Required structure:
+1. **Executive Summary** — with these EXACT metadata values (do NOT invent names):
+ - **Repository:** {repo_name}
+ - **Audit Date:** {audit_date}
+ - **Auditor:** Tooling Agents
+ - **ASVS Requirement:** {asvs} — {asvs_description[:500]}
+ - Files analyzed: {len(relevant_files)} relevant out of {len(filtered_files)} total, {skipped_count} skipped
+ - Finding counts table
+2. **Security Controls Inventory** — each control with location, purpose, coverage status
+3. **Critical File Review** — tables for key files showing all security-sensitive functions reviewed
+4. **Findings** — grouped by severity (Critical → High → Medium → Low), each with:
+ - ID (format: ASVS-{asvs.replace('.', '')}-SEV-NNN)
+ - Location with file, function, line numbers
+ - Related functions checked
+ - Description, vulnerable code, data flow, PoC, impact, remediation
+5. **Positive Security Patterns** — what's done well
+6. **Architecture Observations**
+7. **Recommendations Summary** — Immediate/Short-term/Long-term
+8. **Appendix: Files Analyzed** — collapsible list
+
+Finding counts detected: Critical={findings_count['Critical']}, High={findings_count['High']}, Medium={findings_count['Medium']}, Low={findings_count['Low']}
+
+## Files Analyzed
+{file_list}
+
+## Analysis to Format
+
+{consolidated_analysis}"""
+
+ format_messages = [{"role": "user", "content": format_prompt}]
+ format_tokens = count_message_tokens(format_messages, SONNET_PROVIDER, SONNET_MODEL)
+ sonnet_limit = int(SONNET_CONTEXT * 0.80)
+
+ print(f" Format prompt: {format_tokens} tokens (limit: {sonnet_limit})", flush=True)
+
+ if format_tokens <= sonnet_limit:
+ for attempt in range(2):
+ try:
+ final_report, _ = await call_llm(
+ provider=SONNET_PROVIDER, model=SONNET_MODEL,
+ messages=format_messages, parameters=SONNET_PARAMS,
+ timeout=600,
+ )
+ break
+ except Exception as e:
+ if attempt == 0:
+ print(f" Formatting attempt 1 failed ({type(e).__name__}), retrying...", flush=True)
+ await asyncio.sleep(5)
+ else:
+ print(f" Formatting failed, using raw analysis: {e}", flush=True)
+ final_report = consolidated_analysis
+ else:
+ print(f" Format prompt too large, using raw analysis with header", flush=True)
+ header = f"""# Security Audit Report
+
+## Executive Summary
+
+| Field | Value |
+|-------|-------|
+| **Repository** | {repo_name} |
+| **Audit Date** | {audit_date} |
+| **Auditor** | Tooling Agents |
+| **ASVS Requirement** | {asvs} — {asvs_description[:500]} |
+| **Files Analyzed** | {len(relevant_files)} relevant / {len(filtered_files)} total |
+| **Files Skipped** | {skipped_count} |
+| **Analysis Batches** | {len(opus_batches)} |
+
+### Findings Overview
+
+| Severity | Count |
+|----------|-------|
+| 🔴 Critical | {findings_count['Critical']} |
+| 🟠 High | {findings_count['High']} |
+| 🟡 Medium | {findings_count['Medium']} |
+| 🟢 Low | {findings_count['Low']} |
+
+---
+
+"""
+ appendix = f"""
+
+---
+
+## Appendix: Files Analyzed
+
+<details>
+<summary>Click to expand ({len(relevant_files)} files)</summary>
+
+{file_list}
+
+</details>
+"""
+ final_report = header + consolidated_analysis + appendix
+
+ # Save report
+ report_ns = data_store.use_namespace("audit-reports")
+ report_key = f"asvs-{asvs}-{'-'.join(namespaces)}"
+ report_ns.set(report_key, {
+ "asvs": asvs,
+ "namespaces": namespaces,
+ "files_analyzed": len(relevant_files),
+ "files_total": len(filtered_files),
+ "files_skipped": skipped_count,
+ "findings": findings_count,
+ "report": final_report[:50000],
+ })
+
+ print(f"\n=== Done: {len(final_report)} chars ===", flush=True)
+ return {"outputText": final_report}
+
+ finally:
+ await http_client.aclose()
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org