You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/05/02 22:19:35 UTC
(tooling-agents) branch main updated: ASVS load
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
The following commit(s) were added to refs/heads/main by this push:
new c58e2b6 ASVS load
c58e2b6 is described below
commit c58e2b678d8faf4e0c9d18022ff85ed572e540f0
Author: Andrew Musselman <an...@gmail.com>
AuthorDate: Sat May 2 15:19:16 2026 -0700
ASVS load
---
ASVS/README.md | 58 ++++---
ASVS/agents/asvs_load_data.py | 351 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 385 insertions(+), 24 deletions(-)
diff --git a/ASVS/README.md b/ASVS/README.md
index 96f27d4..e01dd1e 100644
--- a/ASVS/README.md
+++ b/ASVS/README.md
@@ -7,18 +7,18 @@ Built on [Gofannon](https://github.com/The-AI-Alliance/gofannon) — see [docs/g
## Pipeline Flow
```
-orchestrate_asvs_audit_to_github (single entry point)
+asvs_orchestrate (single entry point)
│
- ├──▶ download_github_repo_to_datastore (once — fetches source code)
+ ├──▶ asvs_download_repo (once — fetches source code)
│
- ├──▶ discover_codebase_architecture (once — generates audit plan)
+ ├──▶ asvs_discover (once — generates audit plan)
│
- ├──▶ run_asvs_security_audit (× N sections)
- │ └──▶ add_markdown_file_to_github_directory (× N)
+ ├──▶ asvs_audit (× N sections)
+ │ └──▶ asvs_push_github (× N)
│
- ├──▶ consolidate_asvs_security_audit_reports (once — final report)
+ ├──▶ asvs_consolidate (once — final report)
│
- └──▶ redact + publish (if privateRepo is set)
+ └──▶ redact + publish (if privateRepo is set)
├──▶ read full reports from private repo
├──▶ strip Critical findings
├──▶ push redacted reports to public repo
@@ -31,7 +31,7 @@ Pre-requisites (one-time, outside the pipeline):
## Quick Start
-Run `orchestrate_asvs_audit_to_github` with:
+Run `asvs_orchestrate` with:
| Input | Value |
|---|---|
@@ -120,7 +120,7 @@ When `privateRepo` is absent, all reports go directly to `outputRepo` (current b
All gofannon inputs are strings.
-### 1. orchestrate_asvs_audit_to_github
+### 1. asvs_orchestrate
The main entry point. Calls all other agents.
@@ -133,7 +133,7 @@ The main entry point. Calls all other agents.
| `outputToken` | yes | PAT with write access to output repo |
| `outputDirectory` | yes | Base directory — repo name and commit hash are appended automatically |
| `discover` | no | `"true"` or `"false"` (default `"true"`) |
-| `level` | no | `"L1"`, `"L2"`, or `"L3"` (default `"L3"` — use `"L1"` for a quick baseline audit) |
+| `level` | no | `"L1"`, `"L2"`, or `"L3"` (default empty, treated as L3 — use `"L1"` for a quick baseline audit) |
| `severityThreshold` | no | `"CRITICAL"`, `"HIGH"`, `"MEDIUM"`, or empty |
| `consolidate` | no | `"true"` or `"false"` (default `"true"`) |
| `privateRepo` | no | Private repo for full unredacted reports (enables carve-out) |
@@ -145,15 +145,15 @@ The main entry point. Calls all other agents.
**Namespace derivation:** The orchestrator parses `sourceRepo` to derive the code namespace automatically. `apache/airflow` → `files:apache/airflow`. If `supplementalData` is `audit_guidance`, the namespace list becomes `["files:apache/airflow", "audit_guidance"]`.
**Agents called:**
-1. `download_github_repo_to_datastore` — downloads source code
-2. `discover_codebase_architecture` — generates audit plan (if `discover="true"`)
-3. `run_asvs_security_audit` — once per ASVS section
-4. `add_markdown_file_to_github_directory` — once per section, plus consolidated/issues
-5. `consolidate_asvs_security_audit_reports` — final report (if `consolidate="true"`)
+1. `asvs_download_repo` — downloads source code
+2. `asvs_discover` — generates audit plan (if `discover="true"`)
+3. `asvs_audit` — once per ASVS section
+4. `asvs_push_github` — once per section, plus consolidated/issues
+5. `asvs_consolidate` — final report (if `consolidate="true"`)
---
-### 2. download_github_repo_to_datastore
+### 2. asvs_download_repo
Downloads a GitHub repo (or subdirectory) into the data store.
@@ -167,7 +167,7 @@ Stores files in namespace `files:{owner}/{repo}`. When a path prefix is included
---
-### 3. discover_codebase_architecture
+### 3. asvs_discover
Scans codebase, generates domains + file lists + false positive guidance.
@@ -179,7 +179,7 @@ Scans codebase, generates domains + file lists + false positive guidance.
---
-### 4. run_asvs_security_audit
+### 4. asvs_audit
Audits code against a single ASVS requirement.
@@ -191,18 +191,20 @@ JSON fields inside `inputText`:
| Field | Required | Description |
|---|---|---|
-| `namespaces` | yes | Array of data store namespaces |
-| `asvs` | yes | ASVS section (e.g., `"6.1.1"`) |
+| `namespaces` | yes | Array of data store namespaces (singular `namespace` also accepted) |
+| `asvs` | yes | ASVS section (e.g., `"6.1.1"`) — alias `asvs_section` also accepted |
| `includeFiles` | no | Array of file glob patterns — skips relevance filtering |
| `domainContext` | no | Architecture context for Opus prompt |
| `severityThreshold` | no | Minimum severity to report |
| `falsePositiveGuidance` | no | Array of patterns to suppress |
+If `inputText` isn't valid JSON, the agent falls back to regex-parsing `namespace:` and `asvs:` keys out of free-form text.
+
**Output:** `outputText` — markdown audit report.
---
-### 5. consolidate_asvs_security_audit_reports
+### 5. asvs_consolidate
Reads per-section reports from GitHub, deduplicates, produces consolidated report + issues.
@@ -217,14 +219,22 @@ Reads per-section reports from GitHub, deduplicates, produces consolidated repor
---
-### 6. add_markdown_file_to_github_directory
+### 6. asvs_push_github
| Input | Required | Description |
|---|---|---|
-| `inputText` | yes | JSON with `repo`, `token`, `directory`, `filename` |
-| `commitMessage` | yes | Git commit message |
+| `inputText` | yes | JSON with `repo`, `token`, `directory`, `filename` (see optional fields below) |
+| `commitMessage` | no | Git commit message (default: `"Add markdown file"`) |
| `fileContents` | yes | Markdown content |
+Optional fields inside `inputText`:
+
+| Field | Description |
+|---|---|
+| `branch` | Target branch (defaults to the repo's default branch) |
+| `filePath` | Full repo-relative path as an alternative to `directory` + `filename` |
+| `apiBase` | GitHub API base URL (default `https://api.github.com`; set for GitHub Enterprise) |
+
**Output:** `outputText` — GitHub API response.
---
diff --git a/ASVS/agents/asvs_load_data.py b/ASVS/agents/asvs_load_data.py
new file mode 100644
index 0000000..224d4e7
--- /dev/null
+++ b/ASVS/agents/asvs_load_data.py
@@ -0,0 +1,351 @@
+# asvs_load_data
+
+from agent_factory.remote_mcp_client import RemoteMCPClient
+import httpx
+
+async def run(input_dict, tools):
+ mcpc = { url : RemoteMCPClient(remote_url = url) for url in tools.keys() }
+ http_client = httpx.AsyncClient(timeout=60.0)
+ try:
+ import csv
+ import io
+ import re
+ import asyncio
+
+ # =============================================================
+ # Read inputs from their own fields. Anything in inputText is
+ # also accepted as a fallback for backwards compatibility.
+ # =============================================================
+ def _coerce_bool(v, default=False):
+ if isinstance(v, bool):
+ return v
+ if v is None:
+ return default
+ return str(v).strip().lower() in ("true", "1", "yes", "y", "on")
+
+ TAG_RE = re.compile(r"^v?\d+\.\d+\.\d+$")
+
+ def _normalize_tag(s):
+ if s is None:
+ return None
+ s = str(s).strip()
+ if not s or not TAG_RE.match(s):
+ return None
+ return s if s.startswith("v") else f"v{s}"
+
+ # Primary: discrete fields (which is how this agent's UI is wired).
+ version_raw = input_dict.get("version") or input_dict.get("tag") or ""
+ clear_raw = input_dict.get("clear")
+ if clear_raw is None:
+ clear_raw = input_dict.get("clearExisting")
+ token_raw = input_dict.get("githubToken") or input_dict.get("token") or ""
+ enrich_raw = input_dict.get("enrichMarkdown")
+
+ # Fallback: parse inputText for the same keys (one per line, k: v).
+ input_text = (input_dict.get("inputText") or "").strip()
+ if input_text:
+ for raw_line in input_text.split("\n"):
+ line = raw_line.strip()
+ if not line or ":" not in line:
+ continue
+ k, _, v = line.partition(":")
+ k = k.strip().lower()
+ v = v.strip()
+ if k in ("version", "tag") and not version_raw:
+ version_raw = v
+ elif k in ("clear", "clear_existing", "clearexisting") and clear_raw is None:
+ clear_raw = v
+ elif k in ("token", "github_token", "githubtoken") and not token_raw:
+ token_raw = v
+ elif k in ("enrich_markdown", "enrichmarkdown", "markdown") and enrich_raw is None:
+ enrich_raw = v
+
+ tag = _normalize_tag(version_raw) or "v5.0.0"
+ if version_raw and _normalize_tag(version_raw) is None:
+ return {"outputText": (
+ f"Error: invalid version '{version_raw}' (expected vN.N.N or N.N.N)"
+ )}
+ clear_existing = _coerce_bool(clear_raw, default=False)
+ github_token = (token_raw or "").strip()
+ enrich_markdown = _coerce_bool(enrich_raw, default=True)
+
+ ver = tag.lstrip("v")
+ if not ver.startswith("5."):
+ return {"outputText": (
+ f"Error: only ASVS v5.x is supported (got {tag}). "
+ f"v4 has a different file layout."
+ )}
+
+ csv_url = (
+ f"https://raw.githubusercontent.com/OWASP/ASVS/{tag}/5.0/docs_en/"
+ f"OWASP_Application_Security_Verification_Standard_{ver}_en.csv"
+ )
+
+ gh_headers = {
+ "Accept": "application/vnd.github+json",
+ "X-GitHub-Api-Version": "2022-11-28",
+ }
+ if github_token:
+ gh_headers["Authorization"] = f"Bearer {github_token}"
+
+ print(f"Loading ASVS {tag}", flush=True)
+ print(f" CSV: {csv_url}", flush=True)
+ print(f" clear_existing: {clear_existing}", flush=True)
+ print(f" enrich_markdown: {enrich_markdown}", flush=True)
+
+ # =============================================================
+ # Fetch + validate CSV
+ # =============================================================
+ csv_resp = await http_client.get(csv_url, follow_redirects=False)
+ if csv_resp.status_code != 200:
+ return {"outputText": (
+ f"Error fetching CSV ({csv_resp.status_code}): {csv_url}\n"
+ f"Tag '{tag}' may not exist, or the CSV may not be at the expected path "
+ f"in this version. Check https://github.com/OWASP/ASVS/tree/{tag}/5.0/docs_en\n"
+ f"Response body (first 500 chars): {csv_resp.text[:500]}"
+ )}
+
+ csv_text = csv_resp.content.decode("utf-8-sig", errors="replace")
+
+ EXPECTED_HEADER = "chapter_id,chapter_name,section_id,section_name,req_id,req_description,L"
+ first_line = csv_text.splitlines()[0] if csv_text else ""
+ if first_line.strip() != EXPECTED_HEADER:
+ return {"outputText": (
+ f"Error: response from {csv_url} is not the expected ASVS CSV.\n"
+ f"Expected header:\n {EXPECTED_HEADER}\n"
+ f"Got first line:\n {first_line[:200]}"
+ )}
+
+ # =============================================================
+ # Parse CSV
+ # =============================================================
+ chapters = {}
+ sections = {}
+ requirements = {}
+
+ reader = csv.DictReader(io.StringIO(csv_text))
+ rows_seen = 0
+ for row in reader:
+ rows_seen += 1
+ chapter_id = row["chapter_id"].lstrip("V").strip()
+ section_id = row["section_id"].lstrip("V").strip()
+ req_id = row["req_id"].lstrip("V").strip()
+ try:
+ level = int(row["L"])
+ except (ValueError, KeyError):
+ level = 1
+
+ chapters.setdefault(chapter_id, {
+ "chapter_id": chapter_id,
+ "chapter_name": row["chapter_name"].strip(),
+ "control_objective": "",
+ })
+ sections.setdefault(section_id, {
+ "section_id": section_id,
+ "section_name": row["section_name"].strip(),
+ "chapter_id": chapter_id,
+ "description": "",
+ })
+ requirements[req_id] = {
+ "req_id": req_id,
+ "req_description": row["req_description"].strip(),
+ "level": level,
+ "section_id": section_id,
+ "chapter_id": chapter_id,
+ }
+
+ print(f" Parsed: {rows_seen} rows -> {len(chapters)} chapters, "
+ f"{len(sections)} sections, {len(requirements)} requirements", flush=True)
+
+ if not requirements:
+ return {"outputText": f"Error: CSV parsed but produced 0 requirements. URL: {csv_url}"}
+
+ # =============================================================
+ # Markdown enrichment for control_objective and section descriptions.
+ # The CSV doesn't carry these and the JSON file only carries
+ # structural metadata; chapter intros and section blurbs live only
+ # in the per-chapter markdown files at 5.0/en/0x*-V*-*.md.
+ # =============================================================
+ ch_enriched = 0
+ sec_enriched = 0
+ md_skipped_reason = None
+
+ if enrich_markdown:
+ api_base = f"https://api.github.com/repos/OWASP/ASVS"
+ print(f" Listing chapter markdown files...", flush=True)
+ # Markdown lives in 5.0/en/ — the docs_en/ folder is for
+ # generated artifacts (CSV/JSON/PDF/DOCX) only.
+ list_resp = await http_client.get(
+ f"{api_base}/contents/5.0/en?ref={tag}",
+ headers=gh_headers,
+ )
+ if list_resp.status_code != 200:
+ md_skipped_reason = f"contents API returned {list_resp.status_code}"
+ if list_resp.status_code in (403, 429):
+ md_skipped_reason += " (rate-limited; pass githubToken to raise the limit)"
+ print(f" WARNING: {md_skipped_reason}; skipping markdown enrichment "
+ f"(tried 5.0/en/)", flush=True)
+ else:
+ items = list_resp.json()
+ # Match chapter files: 0x10-V1-Encoding.md, 0x11-V2-..., etc.
+ md_re = re.compile(r"^0x[0-9a-fA-F]+-V(\d+)[-.].*\.md$")
+ md_files = []
+ for it in items:
+ if it.get("type") != "file":
+ continue
+ name = it.get("name", "")
+ m = md_re.match(name)
+ if m and it.get("download_url"):
+ md_files.append((int(m.group(1)), name, it["download_url"]))
+ md_files.sort()
+ print(f" Found {len(md_files)} chapter markdown files", flush=True)
+
+ sem = asyncio.Semaphore(4)
+
+ async def _fetch_md(name, url):
+ async with sem:
+ try:
+ r = await http_client.get(url, headers=gh_headers, follow_redirects=True)
+ if r.status_code == 200:
+ return name, r.text
+ print(f" {name}: HTTP {r.status_code}", flush=True)
+ except Exception as e:
+ print(f" {name}: fetch failed ({type(e).__name__}: {e})", flush=True)
+ return name, None
+
+ fetched = await asyncio.gather(*[_fetch_md(n, u) for _, n, u in md_files])
+
+ # =====================================================
+ # Markdown layout (v5):
+ # # V<N> <Chapter Name>
+ # <control objective paragraph(s)>
+ # ## V<N>.<M> <Section Name>
+ # <section description paragraph(s)>
+ # | # | Description | L | <-- requirement table starts
+ # =====================================================
+ chapter_h_re = re.compile(r"^# V(\d+)\s+(.+?)\s*$", re.MULTILINE)
+ section_h_re = re.compile(r"^## V(\d+\.\d+)\s+(.+?)\s*$", re.MULTILINE)
+ table_start_re = re.compile(r"^\s*\|\s*#\s*\|", re.MULTILINE)
+
+ def _clean(text):
+ # Drop a "## Control Objective" sub-header if present.
+ text = re.sub(r"^##\s+Control Objective\s*$", "", text, flags=re.MULTILINE)
+ paragraphs = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
+ return "\n\n".join(paragraphs).strip()
+
+ for name, md_text in fetched:
+ if not md_text:
+ continue
+ ch_m = chapter_h_re.search(md_text)
+ if not ch_m:
+ continue
+ chapter_id = ch_m.group(1)
+
+ sec_matches = list(section_h_re.finditer(md_text))
+
+ # Chapter control objective: between # V<N> and the first ## V<N>.<M>
+ co_start = ch_m.end()
+ co_end = sec_matches[0].start() if sec_matches else len(md_text)
+ co = _clean(md_text[co_start:co_end])
+ if co and chapter_id in chapters:
+ chapters[chapter_id]["control_objective"] = co
+ ch_enriched += 1
+
+ # Section descriptions
+ for i, sm in enumerate(sec_matches):
+ section_id = sm.group(1)
+ sd_start = sm.end()
+ sd_end = sec_matches[i + 1].start() if i + 1 < len(sec_matches) else len(md_text)
+ body = md_text[sd_start:sd_end]
+ tm = table_start_re.search(body)
+ if tm:
+ body = body[:tm.start()]
+ sd = _clean(body)
+ if sd and section_id in sections:
+ sections[section_id]["description"] = sd
+ sec_enriched += 1
+
+ print(f" Markdown enrichment: {ch_enriched}/{len(chapters)} chapters, "
+ f"{sec_enriched}/{len(sections)} sections", flush=True)
+
+ # =============================================================
+ # Write to data store
+ # =============================================================
+ asvs_ns = data_store.use_namespace("asvs")
+
+ cleared_count = 0
+ existing = asvs_ns.list_keys() or []
+ existing_asvs = [k for k in existing if k.startswith("asvs:")]
+ if clear_existing:
+ if existing_asvs:
+ print(f" Clearing {len(existing_asvs)} existing 'asvs:*' keys", flush=True)
+ for k in existing_asvs:
+ asvs_ns.delete(k)
+ cleared_count = len(existing_asvs)
+ else:
+ if existing_asvs:
+ print(f" Skipping clear; {len(existing_asvs)} existing 'asvs:*' keys "
+ f"will be overwritten in place where IDs match", flush=True)
+
+ print(f" Writing {len(chapters)} chapters...", flush=True)
+ for chapter_id, ch in chapters.items():
+ asvs_ns.set(f"asvs:chapters:{chapter_id}", ch)
+
+ print(f" Writing {len(sections)} sections...", flush=True)
+ for section_id, sec in sections.items():
+ asvs_ns.set(f"asvs:sections:{section_id}", sec)
+
+ print(f" Writing {len(requirements)} requirements...", flush=True)
+ for req_id, req in requirements.items():
+ asvs_ns.set(f"asvs:requirements:{req_id}", req)
+
+ # =============================================================
+ # Sanity check
+ # =============================================================
+ sample_req_id = sorted(
+ requirements.keys(), key=lambda s: [int(p) for p in s.split(".")]
+ )[0]
+ sample_req = asvs_ns.get(f"asvs:requirements:{sample_req_id}")
+ sample_sec = asvs_ns.get(f"asvs:sections:{sample_req.get('section_id','')}") if sample_req else None
+ sample_ch = asvs_ns.get(f"asvs:chapters:{sample_req.get('chapter_id','')}") if sample_req else None
+
+ sanity_ok = bool(
+ sample_req and sample_req.get("req_description") and sample_req.get("level")
+ and sample_sec and sample_sec.get("section_name")
+ and sample_ch and sample_ch.get("chapter_name")
+ )
+
+ levels = {1: 0, 2: 0, 3: 0}
+ for r in requirements.values():
+ lv = r.get("level")
+ if lv in levels:
+ levels[lv] += 1
+
+ summary_lines = [
+ f"ASVS {tag} loaded into data_store namespace 'asvs'",
+ f" Source URL: {csv_url}",
+ f" Chapters: {len(chapters)}",
+ f" Sections: {len(sections)}",
+ f" Requirements: {len(requirements)}",
+ f" L1: {levels[1]}",
+ f" L2: {levels[2]}",
+ f" L3: {levels[3]}",
+ f" Cleared first: {cleared_count} existing keys" if clear_existing else " Cleared first: no (clear=false)",
+ f" Chapter objectives populated: {ch_enriched}/{len(chapters)}",
+ f" Section descriptions populated: {sec_enriched}/{len(sections)}",
+ f" Sanity check (read-back of {sample_req_id}): {'OK' if sanity_ok else 'FAILED'}",
+ ]
+ if md_skipped_reason:
+ summary_lines.append(f" Markdown enrichment skipped: {md_skipped_reason}")
+ if sample_req:
+ summary_lines.append(
+ f" Sample req {sample_req_id}: L{sample_req.get('level','?')} — "
+ f"{sample_req.get('req_description','')[:120]}"
+ )
+
+ result = "\n".join(summary_lines)
+ print(result, flush=True)
+ return {"outputText": result}
+
+ finally:
+ await http_client.aclose()
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org