You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/04/23 09:33:53 UTC
(tooling-agents) branch main updated: Multi-spec architecture ASVS -> general purpose
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
The following commit(s) were added to refs/heads/main by this push:
new 67bf39e Multi-spec architecture ASVS -> general purpose
67bf39e is described below
commit 67bf39e9ff336fbd7136ae00495f4ee3670842ad
Author: Andrew Musselman <an...@gmail.com>
AuthorDate: Thu Apr 23 02:33:38 2026 -0700
Multi-spec architecture ASVS -> general purpose
---
docs/roadmap/README.md | 9 +-
docs/roadmap/multi-spec-architecture.md | 331 ++++++++++++++++++++++++++++++++
docs/roadmap/specs/api-top-10.md | 2 +-
docs/roadmap/specs/asf-baseline.md | 4 +-
docs/roadmap/specs/asvs.md | 92 +++++++++
docs/roadmap/specs/cwe-top-25.md | 4 +-
6 files changed, 435 insertions(+), 7 deletions(-)
diff --git a/docs/roadmap/README.md b/docs/roadmap/README.md
index 42f4100..2d5fe5b 100644
--- a/docs/roadmap/README.md
+++ b/docs/roadmap/README.md
@@ -14,20 +14,25 @@ Where ASVS fits (web apps like ATR and Steve) and where it doesn't (libraries, b
Test harness design for measuring pipeline quality and operating at scale. Covers fixtures (known codebases with known vulns), metrics (recall, precision, false positive rate), LLM-as-judge for semantic comparison, auto-filed GitHub issues for novel errors, and operational dashboards. The ATR da901ba L1+L2 run serves as the regression baseline.
+### [Multi-Spec Architecture](multi-spec-architecture.md)
+
+Implementation plan for Phase 0: renaming the pipeline from ASVS to security, making agents spec-agnostic, adding the `spec` input parameter, data store schema per spec, spec selection modes, cross-spec deduplication, and migration path. This is the prerequisite for all spec additions below.
+
### Multi-Spec Expansion
The pipeline currently audits against OWASP ASVS only. These plans extend it to additional security specifications, with automatic spec selection based on project type.
| Phase | Spec | Best For | Effort | Plan |
|---|---|---|---|---|
-| **0** | Rename ASVS → security | All | 3 days | (no doc — architectural refactor) |
+| **Done** | [OWASP ASVS v5.0.0](specs/asvs.md) | Web applications | Complete | In production — ATR and Steve audited |
+| **0** | [Rename ASVS → security](multi-spec-architecture.md) | All | 4 days | Agent renames, `spec` input, cross-spec dedup |
| **1** | [CWE Top 25](specs/cwe-top-25.md) | Libraries, any code | ~5 days | Memory safety, injection, race conditions |
| **1** | [OWASP API Top 10](specs/api-top-10.md) | API-heavy projects | ~2.5 days | IDOR, mass assignment, rate limiting |
| **2** | [ASF Security Baseline](specs/asf-baseline.md) | All ASF projects | ~6 days + review | Release signing, license compliance, ASF auth patterns |
| **2** | [SLSA Build Levels](specs/slsa.md) | Publishing projects | ~6 days | Provenance, reproducibility, artifact signing |
| **3** | Community-contributed | Any | Ongoing | Load requirements into data store, no code changes |
-After Phase 0, adding a new spec requires no agent code changes — just requirements in the data store, an optional prompt template, and an update to the discovery agent's project type mapping.
+After [Phase 0](multi-spec-architecture.md), adding a new spec requires no agent code changes — just requirements in the data store, an optional prompt template, and an update to the discovery agent's project type mapping.
### How Specs Combine
diff --git a/docs/roadmap/multi-spec-architecture.md b/docs/roadmap/multi-spec-architecture.md
new file mode 100644
index 0000000..801fbf4
--- /dev/null
+++ b/docs/roadmap/multi-spec-architecture.md
@@ -0,0 +1,331 @@
+# Multi-Spec Architecture
+
+Implementation plan for expanding the pipeline from ASVS-only to multiple security specifications. This covers the agent renames, data model changes, spec selection logic, and migration path.
+
+## Current Architecture (ASVS-Only)
+
+```
+ASVS/
+├── agents/
+│ ├── orchestrate_asvs_audit_to_github.py
+│ ├── discover_codebase_architecture.py
+│ ├── run_asvs_security_audit.py
+│ └── consolidate_asvs_security_audit_reports.py
+├── audit_guidance/
+└── reports/
+```
+
+Data store: `asvs` namespace with 345 requirements. One spec, one namespace.
+
+## Target Architecture (Multi-Spec)
+
+```
+security/
+├── agents/
+│ ├── orchestrate_security_audit.py
+│ ├── discover_codebase_architecture.py
+│ ├── run_security_audit.py
+│ └── consolidate_security_audit_reports.py
+├── specs/
+│ ├── asvs/ ← OWASP ASVS v5.0.0 (345 requirements)
+│ ├── cwe-top-25/ ← CWE/SANS Top 25 (25 entries)
+│ ├── api-top-10/ ← OWASP API Security Top 10 (10 entries)
+│ ├── slsa/ ← SLSA v1.0 Build Levels
+│ └── asf-baseline/ ← ASF-specific baseline (custom)
+├── audit_guidance/
+└── reports/
+```
+
+Data store: one namespace per spec (`asvs`, `cwe-top-25`, `api-top-10`, `slsa`, `asf-baseline`). Each follows the same schema.
+
+## Phase 0: Rename and Abstract
+
+Before adding any new spec, make the existing pipeline spec-agnostic. This is a backward-compatible rename — the ASVS flow works identically afterward.
+
+### Agent Renames
+
+| Current | New | Deployment aliases |
+|---|---|---|
+| `orchestrate_asvs_audit_to_github` | `orchestrate_security_audit` | Keep old name as alias |
+| `run_asvs_security_audit` | `run_security_audit` | Keep old name as alias |
+| `consolidate_asvs_security_audit_reports` | `consolidate_security_audit_reports` | Keep old name as alias |
+| `discover_codebase_architecture` | unchanged | — |
+| `download_github_repo_to_datastore` | unchanged | — |
+| `add_markdown_file_to_github_directory` | unchanged | — |
+
+Old deployment names stay registered in CouchDB, forwarding to the new agents. Remove after all callers migrate.
+
+### Audit Agent: `spec` Input
+
+The audit agent gains a `spec` field in its JSON input:
+
+```json
+{
+ "namespaces": ["files:apache/steve/v3"],
+ "asvs": "6.1.1",
+ "spec": "asvs"
+}
+```
+
+When `spec` is `"asvs"`, look up the requirement in the `asvs` namespace (current behavior). When `spec` is `"cwe-top-25"`, look up in `cwe-top-25`:
+
+```json
+{
+ "namespaces": ["files:apache/commons-lang"],
+ "asvs": "CWE-79",
+ "spec": "cwe-top-25"
+}
+```
+
+The `asvs` field name is a misnomer at this point — it's really `requirement_id`. Rename it internally but accept both for backward compatibility:
+
+```python
+requirement_id = input_json.get("requirement_id") or input_json.get("asvs", "")
+spec = input_json.get("spec", "asvs")
+```
+
+### Orchestrator: `specs` Input
+
+The orchestrator accepts a `specs` input (string, comma-separated):
+
+| Input | Behavior |
+|---|---|
+| `"asvs"` | Run only ASVS (backward compatible) |
+| `"auto"` | Discovery agent recommends specs based on project type |
+| `"asvs,cwe-top-25"` | Run specific specs |
+| (empty) | Defaults to `"asvs"` initially, `"auto"` later |
+
+The orchestrator loops through specs, loading requirements from each namespace and generating audit tasks:
+
+```python
+for spec in selected_specs:
+ spec_ns = data_store.use_namespace(spec)
+ requirements = spec_ns.list_keys()
+ for req_id in requirements:
+ # Filter by level, language applicability, etc.
+ tasks.append({
+ "spec": spec,
+ "requirement_id": req_id,
+ "namespaces": code_namespaces,
+ })
+```
+
+### Consolidator: Cross-Spec Awareness
+
+The consolidator already groups findings by domain and deduplicates. The enhancement:
+
+1. Group findings by spec in addition to domain
+2. Use `cross_references` from requirement data to merge findings across specs
+3. Add a "Specification Coverage" section to the consolidated report
+
+```python
+# When merging findings:
+if finding_a["spec"] == "asvs" and finding_b["spec"] == "cwe-top-25":
+ # Check cross-references
+ asvs_req = asvs_ns.get(f"asvs:requirements:{finding_a['requirement_id']}")
+ cwe_xrefs = asvs_req.get("cross_references", {}).get("cwe-top-25", [])
+ if finding_b["requirement_id"] in cwe_xrefs:
+ # Same vulnerability from two specs — merge
+ merged = merge_findings(finding_a, finding_b)
+ merged["specs_violated"] = ["asvs", "cwe-top-25"]
+```
+
+## Data Store Schema
+
+Every spec follows the same schema in its namespace. This is what makes the system extensible — the audit agent doesn't need spec-specific code, just a namespace to look up.
+
+```python
+# Namespace: {spec_name}
+# Key: {spec_name}:requirements:{requirement_id}
+
+{
+ "id": "CWE-79", # Unique within this spec
+ "title": "Cross-site Scripting",
+ "description": "The product does not neutralize...",
+ "level": 1, # Maps to L1/L2/L3 filtering
+ "category": "injection", # For domain grouping
+ "languages": ["all"], # Or ["c", "cpp"] for memory safety
+ "spec": "cwe-top-25", # Self-reference
+ "spec_version": "2024",
+ "cross_references": { # For dedup across specs
+ "asvs": ["1.2.1", "1.2.2", "5.1.1"],
+ "api-top-10": ["API3"]
+ }
+}
+```
+
+### Required Fields
+
+| Field | Type | Purpose |
+|---|---|---|
+| `id` | string | Unique requirement identifier |
+| `title` | string | Human-readable name |
+| `description` | string | What to check for |
+| `level` | int (1-3) | Maps to L1/L2/L3 filtering |
+| `spec` | string | Namespace name |
+
+### Optional Fields
+
+| Field | Type | Purpose |
+|---|---|---|
+| `category` | string | Domain grouping hint for discovery agent |
+| `languages` | array | Language applicability filter |
+| `spec_version` | string | Spec version identifier |
+| `cross_references` | object | Map of spec_name → array of requirement_ids |
+| `detection_methods` | array | How to look for this issue |
+| `evidence` | array | What constitutes a pass/fail |
+
+## Spec Selection: Discovery Agent Enhancement
+
+The discovery agent currently outputs domain-to-file mappings. The enhancement adds project classification and spec recommendations:
+
+```python
+# Enhanced discovery output
+{
+ "project_type": "web_app",
+ "languages": ["python"],
+ "frameworks": ["quart", "asfquart"],
+ "has_http_endpoints": true,
+ "has_auth": true,
+ "has_api": true,
+ "publishes_packages": false,
+
+ "recommended_specs": [
+ {
+ "spec": "asvs",
+ "coverage": "full",
+ "reason": "Web app with auth, sessions, endpoints"
+ },
+ {
+ "spec": "api-top-10",
+ "coverage": "full",
+ "reason": "REST API detected"
+ },
+ {
+ "spec": "cwe-top-25",
+ "coverage": "supplement",
+ "reason": "General code quality baseline"
+ }
+ ],
+
+ "not_recommended": [
+ {
+ "spec": "slsa",
+ "reason": "No package publishing detected"
+ }
+ ],
+
+ "passes": [...] # Existing domain/file mapping
+}
+```
+
+### Project Type Detection Signals
+
+| Signal | Detection Method | Project Type |
+|---|---|---|
+| HTTP framework imports | Code scan | web_app |
+| Route decorators (`@app.route`, `@router.get`) | Code scan | web_app |
+| `setup.py` / `pyproject.toml` with package config | File check | library |
+| CLI entry points (`console_scripts`) | Config check | cli_tool |
+| Queue consumers, scheduler imports | Code scan | backend_service |
+| Dockerfile + publish workflow | File + GHA check | publishing |
+
+### Default Spec Mapping
+
+| Project Type | Default Specs |
+|---|---|
+| `web_app` | asvs, api-top-10, asf-baseline |
+| `library` | cwe-top-25, asf-baseline |
+| `cli_tool` | cwe-top-25, asf-baseline |
+| `backend_service` | asvs (partial), cwe-top-25, asf-baseline |
+| `build_tool` | asf-baseline, slsa |
+
+When `specs="auto"`, the orchestrator uses these defaults. The user can always override with explicit spec selection.
+
+## Consolidated Report Changes
+
+### Spec Coverage Section
+
+```markdown
+## Specification Coverage
+
+| Spec | Version | Requirements Audited | Findings | Pass | Fail | N/A |
+|------|---------|---------------------|----------|------|------|-----|
+| ASVS v5.0.0 | 5.0.0 | 253 (L1+L2) | 137 | 89 | 137 | 27 |
+| CWE Top 25 | 2024 | 25 | 12 | 13 | 12 | 0 |
+| API Top 10 | 2023 | 10 | 4 | 6 | 4 | 0 |
+```
+
+### Cross-Spec Findings
+
+```markdown
+### Cross-Spec Findings
+
+12 findings match requirements in multiple specs:
+
+| Finding | ASVS | CWE | API Top 10 |
+|---------|------|-----|------------|
+| FINDING-001: Missing rate limiting | 2.4.1 | — | API4 |
+| FINDING-015: SQL injection in search | 1.2.1 | CWE-89 | API1 |
+| FINDING-023: Missing TLS validation | 12.1.1 | CWE-295 | API7 |
+```
+
+### Finding Format
+
+Each finding gains a `specs_violated` field:
+
+```markdown
+#### FINDING-042: Missing authorization check on GET /api/elections/{id}/votes
+
+**Specs violated:** ASVS 4.1.1, API1
+**Severity:** High
+**Files:** v3/server/pages.py:426
+```
+
+## Migration Path
+
+### Step 1: Deploy New Agents Alongside Old
+
+Register both names in CouchDB deployments:
+
+```
+run_asvs_security_audit → agent_id (existing)
+run_security_audit → same agent_id (new alias)
+```
+
+The agent code accepts both `asvs` and `requirement_id` fields. When `spec` is absent, defaults to `"asvs"`.
+
+### Step 2: Update Callers
+
+- `rerun-sections.sh` → update agent names
+- Orchestrator → already handles the new names
+- Any external scripts calling the API
+
+### Step 3: Remove Old Aliases
+
+Delete old deployment entries from CouchDB:
+
+```bash
+curl -X DELETE -u admin:password \
+ "http://localhost:5984/deployments/run_asvs_security_audit"
+```
+
+### Step 4: Repo Restructure
+
+```bash
+git mv ASVS/ security/
+# Update all READMEs, scripts, docs
+```
+
+## Estimated Effort (Phase 0 Only)
+
+| Task | Effort |
+|---|---|
+| Rename agents, add `spec`/`requirement_id` inputs | 1 day |
+| Update orchestrator to accept `specs` input, loop through specs | 1 day |
+| Update consolidator for cross-spec grouping and dedup | 1 day |
+| Deploy with aliases, test backward compatibility | Half day |
+| Update READMEs, `rerun-sections.sh`, docs | Half day |
+| **Total** | **~4 days** |
+
+After Phase 0, adding each new spec (CWE, API Top 10, etc.) is a data + prompt exercise — see the individual [spec plans](specs/).
\ No newline at end of file
diff --git a/docs/roadmap/specs/api-top-10.md b/docs/roadmap/specs/api-top-10.md
index 85d9d56..05ca4c7 100644
--- a/docs/roadmap/specs/api-top-10.md
+++ b/docs/roadmap/specs/api-top-10.md
@@ -142,7 +142,7 @@ When API Top 10 and ASVS produce overlapping findings, the consolidated report n
| Write ingest script (10 entries, manually curated) | Half day | None |
| Build cross-reference mapping (API ↔ ASVS ↔ CWE) | Half day | CWE ingest |
| Audit prompt template | Half day | None |
-| Add API detection to discovery agent | Half day | Phase 0 rename |
+| Add API detection to discovery agent | Half day | [Phase 0](../multi-spec-architecture.md) |
| Test with ATR (has REST API) | Half day | Ingest script |
| **Total** | **~2.5 days** | |
diff --git a/docs/roadmap/specs/asf-baseline.md b/docs/roadmap/specs/asf-baseline.md
index 5f140fe..8fa700c 100644
--- a/docs/roadmap/specs/asf-baseline.md
+++ b/docs/roadmap/specs/asf-baseline.md
@@ -157,9 +157,9 @@ Draft the initial spec based on existing ASF policy documents, then iterate thro
|---|---|---|
| Draft requirements (from ASF policy docs) | 2 days | Community input |
| Write ingest script | Half day | Draft requirements |
-| Implement static checks (no-LLM fast path) | 2 days | Phase 0 rename |
+| Implement static checks (no-LLM fast path) | 2 days | [Phase 0](../multi-spec-architecture.md) |
| Write audit prompts for LLM-required checks | 1 day | Draft requirements |
| Community review and iteration | 2 weeks | Draft complete |
-| Integrate with discovery agent | Half day | Phase 0 rename |
+| Integrate with discovery agent | Half day | [Phase 0](../multi-spec-architecture.md) |
| **Total (engineering)** | **~6 days** | |
| **Total (with review)** | **~3 weeks** | |
\ No newline at end of file
diff --git a/docs/roadmap/specs/asvs.md b/docs/roadmap/specs/asvs.md
new file mode 100644
index 0000000..762df15
--- /dev/null
+++ b/docs/roadmap/specs/asvs.md
@@ -0,0 +1,92 @@
+# OWASP ASVS v5.0.0
+
+**Status: Implemented and in production.**
+
+This is the founding spec of the pipeline. All other specs build on the architecture established here.
+
+## Overview
+
+The [OWASP Application Security Verification Standard v5.0.0](https://github.com/OWASP/ASVS/blob/v5.0.0/5.0/OWASP_Application_Security_Verification_Standard_5.0.0_en.pdf) defines 345 security requirements across 17 chapters, organized into three levels of increasing rigor. It's designed for web applications with HTTP endpoints, authentication, session management, and user-facing interfaces.
+
+See [ASVS Applicability](../asvs-applicability.md) for where this spec fits well and where it doesn't.
+
+## Implementation
+
+| Component | Status |
+|---|---|
+| Requirements in data store | ✅ `asvs` namespace, 345 entries |
+| Ingest script | ✅ `ingest_asvs_standard` |
+| Audit prompt | ✅ In `run_asvs_security_audit` |
+| Discovery integration | ✅ Domain mapping by ASVS chapter |
+| Consolidation | ✅ Dedup within and across domains |
+| Level filtering (L1/L2/L3) | ✅ Orchestrator `level` input |
+| Severity threshold | ✅ Orchestrator `severityThreshold` input |
+| Audit guidance | ✅ Per-project false positive reduction |
+| Pipeline documentation | ✅ [ASVS/README.md](../../../ASVS/README.md) |
+| QA tooling | ✅ `rerun-sections.sh` |
+
+## Data Store Schema
+
+```
+Namespace: asvs
+Key: asvs:requirements:6.1.1
+
+{
+ "id": "6.1.1",
+ "title": "Password Security",
+ "description": "Verify that user-set passwords are at least 8 characters in length...",
+ "level": 1,
+ "chapter": 6,
+ "section_name": "Credential & Secret Management",
+ "spec": "asvs",
+ "spec_version": "5.0.0"
+}
+```
+
+## Coverage by Level
+
+| Level | Requirements | Scope |
+|---|---|---|
+| L1 | 70 | Critical baseline — highest priority requirements |
+| L2 | 183 | Standard security — defenses against common threats |
+| L3 | 92 | Advanced — complete coverage for high-value targets |
+| **Total** | **345** | |
+
+## Production Runs
+
+| Project | Commit | Level | Sections | Findings | Report |
+|---|---|---|---|---|---|
+| ATR | da901ba | L1+L2 | 253 | 137 consolidated | [consolidated-L1-L2.md](../../../ASVS/reports/tooling-trusted-releases/da901ba/consolidated-L1-L2.md) |
+| Steve v3 | d0aa7e9 | L3 | 340/345 | 235 consolidated | [consolidated.md](../../../ASVS/reports/steve/v3/d0aa7e9/consolidated.md) |
+
+## Cross-References
+
+When additional specs are added, ASVS requirements will gain `cross_references` fields mapping to equivalent requirements in other specs:
+
+```json
+{
+ "id": "1.2.1",
+ "cross_references": {
+ "cwe-top-25": ["CWE-79"],
+ "api-top-10": ["API3"]
+ }
+}
+```
+
+This enables the consolidator to merge findings that flag the same vulnerability from multiple specs.
+
+## Lessons Learned
+
+Building the pipeline against ASVS taught us what the multi-spec architecture needs:
+
+1. **Discovery is essential.** Without architecture-aware domain scoping, the LLM wastes tokens analyzing irrelevant files and produces more false positives.
+
+2. **Audit guidance reduces noise.** LLMs consistently misinterpret certain patterns (reverse proxy auth, framework-specific session handling). Per-project guidance documents and inline `# audit_guidance` comments fix this.
+
+3. **"Not applicable" is a valid result.** Many ASVS sections don't apply to a given codebase. The audit agent needs to say so cleanly instead of hallucinating findings.
+
+4. **Extraction is fragile.** Reports with extensive code blocks cause Sonnet to return malformed JSON or echo report content instead of extracting findings. Anchored regex, balanced brace parsing, and `ast.literal_eval` fallbacks handle most cases. Very large reports occasionally still fail — the consolidated report now links to the raw per-section report when extraction fails.
+
+5. **Consolidation at scale needs batching.** Domain groups with 25+ reports need sub-batching to avoid timeouts. Issue formatting needs batches of ~40 findings to avoid token limit truncation.
+
+These lessons directly informed the [multi-spec architecture](../multi-spec-architecture.md) design.
\ No newline at end of file
diff --git a/docs/roadmap/specs/cwe-top-25.md b/docs/roadmap/specs/cwe-top-25.md
index 16a4401..8185c75 100644
--- a/docs/roadmap/specs/cwe-top-25.md
+++ b/docs/roadmap/specs/cwe-top-25.md
@@ -162,7 +162,7 @@ When both ASVS and CWE Top 25 are selected, the consolidator uses cross-referenc
| Write ingest script for CWE data | 1 day | None |
| Build cross-reference mapping (CWE ↔ ASVS) | 1 day | Ingest script |
| Adapt audit prompt template for CWE format | Half day | None |
-| Add language filtering to discovery agent | Half day | Phase 0 rename |
-| Update consolidator for cross-spec dedup | 1 day | Phase 0 rename |
+| Add language filtering to discovery agent | Half day | [Phase 0](../multi-spec-architecture.md) |
+| Update consolidator for cross-spec dedup | 1 day | [Phase 0](../multi-spec-architecture.md) |
| Test with library fixture | 1 day | Eval framework |
| **Total** | **~5 days** | |
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org