You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/07/09 00:46:29 UTC
(tooling-agents) branch main updated: Fix publishing model ID, batch push, real bernies agent
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-agents.git
The following commit(s) were added to refs/heads/main by this push:
new 6ad8f97 Fix publishing model ID, batch push, real bernies agent
6ad8f97 is described below
commit 6ad8f979af211f2c9e61c601b6be5e1078d2cecc
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Wed Jul 8 17:46:20 2026 -0700
Fix publishing model ID, batch push, real bernies agent
Fix publishing agent model ID: us.anthropic.claude-sonnet-4.6 (dot)
was invalid Bedrock ID. Changed to us.anthropic.claude-sonnet-4-6
(hyphen). Root cause of empty publishing report (0 workflows).
Batch push: orchestrator collects all files during pipeline, pushes
in single commit via GitHub Git Data API. No more per-file commits
or asvs_push_github dependency.
Rename gha_bernies to gha_action_health (GitHub Actions bus factor).
New gha_bernies does project dependency analysis: parses manifests
(pyproject.toml, package.json, pom.xml, Cargo.toml, go.mod, Gemfile),
queries PyPI/npm/crates.io for maintainer counts, flags packages
with low bus factor.
---
gha-review/README.md | 6 +-
gha-review/agents/README.md | 9 +-
.../{gha_bernies.py => gha_action_health.py} | 6 +-
gha-review/agents/gha_bernies.py | 714 +++++++++++++--------
gha-review/agents/gha_orchestrator.py | 175 +++--
gha-review/agents/gha_publishing.py | 11 +-
6 files changed, 603 insertions(+), 318 deletions(-)
diff --git a/gha-review/README.md b/gha-review/README.md
index 06ffe94..31303da 100644
--- a/gha-review/README.md
+++ b/gha-review/README.md
@@ -26,7 +26,8 @@ gha_orchestrator
│
├──▸ gha_publishing_detail (YAML parsing + GitHub API run history)
├──▸ gha_artifact_verify (registry API queries)
- ├──▸ gha_bernies (CI supply chain bus factor)
+ ├──▸ gha_action_health (GitHub Actions dependency health)
+ ├──▸ gha_bernies (project dependency bus factor)
│
├──▸ gha_review (combined risk assessment)
├──▸ gha_brief (executive action plan)
@@ -58,7 +59,8 @@ reports/
├── publishing_detail.md enriched workflow data
├── publishing_risks.md dangerous patterns
├── artifact_verification.md registry cross-reference
-├── bernies.md CI supply chain bus factor
+├── bernies.md project dependency bus factor
+├── action_health.md GitHub Actions dependency health
├── json_export.json machine-readable
├── channels/ per-channel publishing detail
│ ├── pypi.md, npm.md, maven_central.md, docker_hub.md, ...
diff --git a/gha-review/agents/README.md b/gha-review/agents/README.md
index 2a42abd..6ee5d11 100644
--- a/gha-review/agents/README.md
+++ b/gha-review/agents/README.md
@@ -11,7 +11,8 @@
├── gha_security.py Pattern-matching security checks, per-PMC reports
├── gha_publishing_detail.py Enriched YAML parsing + GitHub API run history
├── gha_artifact_verify.py Registry API queries for artifact verification
-├── gha_bernies.py CI supply chain bus factor analysis
+├── gha_action_health.py GitHub Actions dependency bus factor
+├── gha_bernies.py Project dependency bus factor analysis
├── gha_review.py Combined risk assessment report
├── gha_brief.py One-page executive action plan
├── gha_json_export.py Machine-readable structured export
@@ -28,6 +29,7 @@ gha_publishing ──▸ ci-classification:{owner}, ci-report:{owner}
gha_security ────▸ ci-security:{owner}, ci-security-history:{owner}
gha_publishing_detail ──▸ ci-publishing-detail:{owner}
gha_artifact_verify ────▸ ci-artifact-verify:{owner}
+gha_action_health ──────▸ ci-action-health:{owner}
gha_bernies ────────▸ ci-bernies:{owner}
gha_review/brief/json_export ──▸ ci-combined:{owner}
```
@@ -54,6 +56,7 @@ Output:
├── brief.md, review.md, publishing.md, security.md
├── publishing_detail.md, publishing_risks.md, artifact_verification.md
├── bernies.md
+├── action_health.md
├── json_export.json
├── channels/
│ ├── README.md
@@ -73,7 +76,8 @@ Output:
| gha_security | `github_owner` | `redacted_severity` |
| gha_publishing_detail | `github_owner`, `read_pat` | `repos`, `channels` |
| gha_artifact_verify | `github_owner` | `repos`, `channels` |
-| gha_bernies | `github_owner`, `read_pat` | — |
+| gha_action_health | `github_owner`, `read_pat` | — |
+| gha_bernies | `github_owner`, `read_pat` | `repos` |
| gha_review, gha_brief, gha_json_export | `github_owner` | `redacted_severity` |
### Multi-File Output Routing
@@ -145,6 +149,7 @@ The orchestrator uses `asvs_push_github` with `inputText` JSON containing `repo`
| `ci-security-history:{owner}` | gha_security | gha_security |
| `ci-publishing-detail:{owner}` | gha_publishing_detail | gha_artifact_verify |
| `ci-artifact-verify:{owner}` | gha_artifact_verify | — |
+| `ci-action-health:{owner}` | gha_action_health | — |
| `ci-bernies:{owner}` | gha_bernies | — |
| `ci-combined:{owner}` | gha_review, gha_brief, gha_json_export | — |
diff --git a/gha-review/agents/gha_bernies.py b/gha-review/agents/gha_action_health.py
similarity index 98%
copy from gha-review/agents/gha_bernies.py
copy to gha-review/agents/gha_action_health.py
index daa82ba..66b5a7f 100644
--- a/gha-review/agents/gha_bernies.py
+++ b/gha-review/agents/gha_action_health.py
@@ -1,5 +1,5 @@
"""
-gha_bernies
+gha_action_health
Identifies GitHub Actions dependencies with low bus factor: few active
maintainers, abandoned repos, or single-contributor dominance. Named
@@ -317,8 +317,8 @@ async def run(input_dict, tools):
report = "\n".join(lines)
# Store in CouchDB
- bernies_ns = data_store.use_namespace(f"ci-bernies:{github_owner}")
- bernies_ns.set_many({
+ action_health_ns = data_store.use_namespace(f"ci-action-health:{github_owner}")
+ action_health_ns.set_many({
"latest_report": report,
"latest_data": json.dumps({
'generated_at': datetime.now(timezone.utc).isoformat(),
diff --git a/gha-review/agents/gha_bernies.py b/gha-review/agents/gha_bernies.py
index daa82ba..5de6476 100644
--- a/gha-review/agents/gha_bernies.py
+++ b/gha-review/agents/gha_bernies.py
@@ -1,18 +1,17 @@
"""
gha_bernies
-Identifies GitHub Actions dependencies with low bus factor: few active
-maintainers, abandoned repos, or single-contributor dominance. Named
-after the "Weekend at Bernies" pattern where a project appears alive
-but its maintainers have moved on.
+Identifies project dependencies with low bus factor: packages maintained
+by too few people, abandoned upstream, or single-maintainer dominated.
-Reads action references from ci-publishing-detail or ci-workflows,
-deduplicates to unique third-party action repos, and queries the
-GitHub API for contributor and commit recency data.
+Reads manifest files (pyproject.toml, package.json, pom.xml, Cargo.toml,
+go.mod) from repos via GitHub API, parses dependencies, and queries
+package registries for maintainer/contributor data.
Inputs:
github_owner — org to analyze (e.g. "apache")
read_pat — GitHub PAT for API calls
+ repos — optional comma-separated filter
"""
from agent_factory.remote_mcp_client import RemoteMCPClient
@@ -32,303 +31,484 @@ async def run(input_dict, tools):
github_owner = input_dict.get("github_owner", "apache")
read_pat = input_dict.get("read_pat", "")
-
- print(f"Bernies analysis for {github_owner}", flush=True)
+ repos_filter = [r.strip() for r in input_dict.get("repos", "").split(",") if r.strip()]
if not read_pat:
- return {"outputText": json.dumps({
- "error": "read_pat is required for GitHub API calls"})}
+ return {"outputText": json.dumps({"error": "read_pat required"})}
+
+ print(f"Bernies dependency analysis for {github_owner}", flush=True)
github_headers = {
"Accept": "application/vnd.github+json",
"Authorization": f"Bearer {read_pat}",
}
- # ASF-exempt orgs — actions from these don't need Bernie checks
- EXEMPT_ORGS = {"actions", "github", "apache"}
-
- # ── Collect all action references from cached YAML ──
- wf_ns = data_store.use_namespace(f"ci-workflows:{github_owner}")
- all_keys = wf_ns.list_keys()
- prefetch_keys = [k for k in all_keys if k.startswith("__prefetch__:")]
- prefetch_data = wf_ns.get_many(prefetch_keys) if prefetch_keys else {}
+ # ── Manifest file patterns and parsers ──
+
+ MANIFEST_SPECS = {
+ 'python': {
+ 'files': ['pyproject.toml', 'requirements.txt', 'setup.cfg'],
+ 'ecosystem': 'pypi',
+ },
+ 'node': {
+ 'files': ['package.json'],
+ 'ecosystem': 'npm',
+ },
+ 'java': {
+ 'files': ['pom.xml'],
+ 'ecosystem': 'maven',
+ },
+ 'rust': {
+ 'files': ['Cargo.toml'],
+ 'ecosystem': 'crates_io',
+ },
+ 'go': {
+ 'files': ['go.mod'],
+ 'ecosystem': 'go',
+ },
+ 'ruby': {
+ 'files': ['Gemfile'],
+ 'ecosystem': 'rubygems',
+ },
+ }
- # Build YAML key list from prefetch metadata
- yaml_keys = []
- for pk, meta in prefetch_data.items():
- repo_name = pk.replace("__prefetch__:", "")
- if not meta or not isinstance(meta, dict) or not meta.get("complete"):
- continue
- for wf in meta.get("workflows", []):
- yaml_keys.append(f"{repo_name}/{wf}")
-
- all_yaml = wf_ns.get_many(yaml_keys) if yaml_keys else {}
- print(f" Read {len(all_yaml)} workflow files", flush=True)
-
- # Parse all uses: lines across all workflows
- # action_repo -> set of (asf_repo, wf_file) that use it
- action_users = defaultdict(set) # "owner/repo" -> {(asf_repo, wf_file)}
- action_versions = defaultdict(set) # "owner/repo" -> {"v1", "abc123", ...}
-
- for yaml_key, content in all_yaml.items():
- if not content or not isinstance(content, str):
- continue
- asf_repo = yaml_key.split("/", 1)[0]
- wf_file = yaml_key.split("/", 1)[1] if "/" in yaml_key else yaml_key
-
- for m in re.finditer(r'uses:\s*([^@\s]+)@([^\s#]+)', content):
- action_ref = m.group(1).strip()
- version = m.group(2).strip()
-
- # Parse owner/repo from the action reference
- parts = action_ref.split("/")
- if len(parts) < 2:
+ def parse_python_deps(content, filename):
+ deps = set()
+ if filename == 'requirements.txt':
+ for line in content.split('\n'):
+ line = line.strip()
+ if line and not line.startswith('#') and not line.startswith('-'):
+ pkg = re.split(r'[>=<!~\[]', line)[0].strip()
+ if pkg:
+ deps.add(pkg.lower())
+ elif filename == 'pyproject.toml':
+ in_deps = False
+ for line in content.split('\n'):
+ if re.match(r'\[project\]|\[tool\.poetry\.dependencies\]', line):
+ in_deps = True
+ continue
+ if line.startswith('[') and in_deps:
+ in_deps = False
+ continue
+ if in_deps:
+ m = re.match(r'dependencies\s*=\s*\[', line)
+ if m:
+ in_deps = True
+ continue
+ m = re.match(r'"([^"]+)"', line.strip())
+ if m:
+ pkg = re.split(r'[>=<!~]', m.group(1))[0].strip()
+ if pkg:
+ deps.add(pkg.lower())
+ m = re.match(r'(\w[\w-]*)\s*=', line.strip())
+ if m and not line.strip().startswith('['):
+ deps.add(m.group(1).lower())
+ elif filename == 'setup.cfg':
+ in_deps = False
+ for line in content.split('\n'):
+ if line.strip() == 'install_requires =':
+ in_deps = True
+ continue
+ if in_deps:
+ if line and not line[0].isspace():
+ break
+ pkg = re.split(r'[>=<!~\[]', line.strip())[0].strip()
+ if pkg:
+ deps.add(pkg.lower())
+ return deps
+
+ def parse_node_deps(content):
+ deps = set()
+ try:
+ data = json.loads(content)
+ for section in ('dependencies', 'devDependencies', 'peerDependencies'):
+ for pkg in data.get(section, {}):
+ if not pkg.startswith('file:') and not pkg.startswith('.'):
+ deps.add(pkg)
+ except Exception:
+ pass
+ return deps
+
+ def parse_java_deps(content):
+ deps = set()
+ for m in re.finditer(r'<dependency>\s*<groupId>([^<]+)</groupId>\s*<artifactId>([^<]+)</artifactId>',
+ content, re.DOTALL):
+ group = m.group(1).strip()
+ artifact = m.group(2).strip()
+ if not group.startswith('${') and not artifact.startswith('${'):
+ deps.add(f"{group}:{artifact}")
+ return deps
+
+ def parse_rust_deps(content):
+ deps = set()
+ in_deps = False
+ for line in content.split('\n'):
+ if re.match(r'\[(.*dependencies.*)\]', line):
+ in_deps = True
continue
- action_owner = parts[0]
- action_repo = parts[1]
-
- # Skip exempt orgs
- if action_owner.lower() in EXEMPT_ORGS:
+ if line.startswith('[') and in_deps:
+ in_deps = False
+ continue
+ if in_deps:
+ m = re.match(r'(\w[\w-]*)\s*=', line)
+ if m:
+ deps.add(m.group(1))
+ return deps
+
+ def parse_go_deps(content):
+ deps = set()
+ in_require = False
+ for line in content.split('\n'):
+ if line.strip() == 'require (':
+ in_require = True
+ continue
+ if line.strip() == ')' and in_require:
+ in_require = False
continue
+ if in_require:
+ parts = line.strip().split()
+ if parts and not parts[0].startswith('//'):
+ deps.add(parts[0])
+ return deps
+
+ def parse_ruby_deps(content):
+ deps = set()
+ for m in re.finditer(r"gem\s+['\"]([^'\"]+)['\"]", content):
+ deps.add(m.group(1))
+ return deps
+
+ PARSERS = {
+ 'pyproject.toml': lambda c: parse_python_deps(c, 'pyproject.toml'),
+ 'requirements.txt': lambda c: parse_python_deps(c, 'requirements.txt'),
+ 'setup.cfg': lambda c: parse_python_deps(c, 'setup.cfg'),
+ 'package.json': parse_node_deps,
+ 'pom.xml': parse_java_deps,
+ 'Cargo.toml': parse_rust_deps,
+ 'go.mod': parse_go_deps,
+ 'Gemfile': parse_ruby_deps,
+ }
- action_full = f"{action_owner}/{action_repo}"
- action_users[action_full].add((asf_repo, wf_file))
- action_versions[action_full].add(version)
+ # ── Registry API queries for maintainer data ──
+
+ async def check_pypi(pkg):
+ try:
+ r = await http_client.get(f"https://pypi.org/pypi/{pkg}/json")
+ if r.status_code != 200:
+ return None
+ data = r.json()
+ info = data.get('info', {})
+ maintainers = info.get('maintainer', '') or ''
+ author = info.get('author', '') or ''
+ maintainer_email = info.get('maintainer_email', '') or ''
+ author_email = info.get('author_email', '') or ''
+ # Count distinct people
+ people = set()
+ for field in [maintainers, author]:
+ for name in field.split(','):
+ name = name.strip()
+ if name and name.lower() not in ('none', ''):
+ people.add(name.lower())
+ return {
+ 'name': pkg,
+ 'ecosystem': 'pypi',
+ 'version': info.get('version', '?'),
+ 'maintainer_count': len(people) if people else 1,
+ 'maintainers': list(people)[:5],
+ 'home_page': info.get('home_page', '') or info.get('project_url', ''),
+ 'source_repo': None,
+ }
+ except Exception:
+ return None
+
+ async def check_npm(pkg):
+ try:
+ r = await http_client.get(f"https://registry.npmjs.org/{pkg}")
+ if r.status_code != 200:
+ return None
+ data = r.json()
+ maintainers = data.get('maintainers', [])
+ latest = data.get('dist-tags', {}).get('latest', '?')
+ repo_url = data.get('repository', {})
+ if isinstance(repo_url, dict):
+ repo_url = repo_url.get('url', '')
+ return {
+ 'name': pkg,
+ 'ecosystem': 'npm',
+ 'version': latest,
+ 'maintainer_count': len(maintainers),
+ 'maintainers': [m.get('name', '?') for m in maintainers[:5]],
+ 'source_repo': repo_url,
+ }
+ except Exception:
+ return None
+
+ async def check_crates(pkg):
+ try:
+ r = await http_client.get(f"https://crates.io/api/v1/crates/{pkg}",
+ headers={"User-Agent": "apache-ci-scan/1.0"})
+ if r.status_code != 200:
+ return None
+ data = r.json()
+ crate = data.get('crate', {})
+ # Get owners
+ owners_resp = await http_client.get(
+ f"https://crates.io/api/v1/crates/{pkg}/owners",
+ headers={"User-Agent": "apache-ci-scan/1.0"})
+ owners = []
+ if owners_resp.status_code == 200:
+ owners = owners_resp.json().get('users', [])
+ return {
+ 'name': pkg,
+ 'ecosystem': 'crates.io',
+ 'version': crate.get('newest_version', '?'),
+ 'maintainer_count': len(owners),
+ 'maintainers': [o.get('login', '?') for o in owners[:5]],
+ 'source_repo': crate.get('repository', ''),
+ }
+ except Exception:
+ return None
+
+ REGISTRY_CHECKS = {
+ 'pypi': check_pypi,
+ 'npm': check_npm,
+ 'crates_io': check_crates,
+ }
- unique_actions = sorted(action_users.keys())
- print(f" Found {len(unique_actions)} unique third-party action repos", flush=True)
+ # ── Discover repos from prefetch ──
- # ── Query GitHub API for each action repo (concurrent) ──
+ wf_ns = data_store.use_namespace(f"ci-workflows:{github_owner}")
+ all_keys = wf_ns.list_keys()
+ prefetch_keys = [k for k in all_keys if k.startswith("__prefetch__:")]
+ prefetch_data = wf_ns.get_many(prefetch_keys) if prefetch_keys else {}
+
+ # Get extras (has file tree info)
+ extras_keys = [k for k in all_keys if k.startswith("__extras__:")]
+ extras_data = wf_ns.get_many(extras_keys) if extras_keys else {}
+
+ repo_names = []
+ for pk, meta in sorted(prefetch_data.items()):
+ name = pk.replace("__prefetch__:", "")
+ if meta and isinstance(meta, dict) and meta.get("complete"):
+ if not repos_filter or name in repos_filter:
+ repo_names.append(name)
+
+ # Only check repos that publish (from publishing detail)
+ detail_ns = data_store.use_namespace(f"ci-publishing-detail:{github_owner}")
+ detail_data = None
+ try:
+ detail_data = detail_ns.get("latest_data")
+ except Exception:
+ pass
+
+ publishing_repos = set()
+ if detail_data and isinstance(detail_data, dict):
+ for repo in detail_data.get('repos', []):
+ publishing_repos.add(repo.get('repo', '').replace(f'{github_owner}/', ''))
+
+ # If we have publishing data, only check those repos
+ if publishing_repos:
+ target_repos = [r for r in repo_names if r in publishing_repos]
+ print(f" Checking {len(target_repos)} publishing repos (of {len(repo_names)} total)", flush=True)
+ else:
+ target_repos = repo_names
+ print(f" No publishing data — checking all {len(target_repos)} repos", flush=True)
+
+ # ── Fetch manifest files and parse dependencies ──
sem = asyncio.Semaphore(10)
- action_health = {} # action_full -> health dict
+ all_deps = defaultdict(set) # ecosystem -> set of package names
+ repo_deps = {} # repo -> {ecosystem: [deps]}
- async def check_action(action_full):
+ async def fetch_manifest(repo_name, manifest_file):
async with sem:
- health = {
- 'action': action_full,
- 'last_commit': None,
- 'last_commit_days_ago': None,
- 'total_contributors': 0,
- 'recent_contributors': 0, # active in past year
- 'top_contributor_share': 0.0,
- 'top_contributor': None,
- 'archived': False,
- 'risk': 'unknown',
- 'error': None,
- }
-
try:
- # Repo metadata (archived status)
- repo_resp = await http_client.get(
- f"https://api.github.com/repos/{action_full}",
- headers=github_headers)
- if repo_resp.status_code == 404:
- health['error'] = 'repo not found'
- health['risk'] = 'critical'
- return action_full, health
- if repo_resp.status_code != 200:
- health['error'] = f'HTTP {repo_resp.status_code}'
- return action_full, health
-
- repo_data = repo_resp.json()
- health['archived'] = repo_data.get('archived', False)
-
- # Last commit
- commits_resp = await http_client.get(
- f"https://api.github.com/repos/{action_full}/commits?per_page=1",
- headers=github_headers)
- if commits_resp.status_code == 200:
- commits = commits_resp.json()
- if commits:
- date_str = commits[0].get('commit', {}).get('committer', {}).get('date', '')
- if date_str:
- health['last_commit'] = date_str[:10]
- try:
- last_dt = datetime.fromisoformat(date_str.replace('Z', '+00:00'))
- days = (datetime.now(timezone.utc) - last_dt).days
- health['last_commit_days_ago'] = days
- except Exception:
- pass
-
- # Contributors (top 30)
- contribs_resp = await http_client.get(
- f"https://api.github.com/repos/{action_full}/contributors?per_page=30&anon=false",
- headers=github_headers)
- if contribs_resp.status_code == 200:
- contribs = contribs_resp.json()
- if isinstance(contribs, list):
- health['total_contributors'] = len(contribs)
- total_commits = sum(c.get('contributions', 0) for c in contribs)
- if contribs and total_commits > 0:
- top = contribs[0]
- health['top_contributor'] = top.get('login', '?')
- health['top_contributor_share'] = round(
- top.get('contributions', 0) / total_commits * 100, 1)
-
- # Recent contributors (commits in past year)
- since = datetime.now(timezone.utc).replace(year=datetime.now().year - 1).isoformat()
- recent_resp = await http_client.get(
- f"https://api.github.com/repos/{action_full}/commits?per_page=100&since={since}",
- headers=github_headers)
- if recent_resp.status_code == 200:
- recent_commits = recent_resp.json()
- if isinstance(recent_commits, list):
- recent_authors = set()
- for c in recent_commits:
- author = c.get('author')
- if author and author.get('login'):
- login = author['login']
- if not login.endswith('[bot]'):
- recent_authors.add(login)
- health['recent_contributors'] = len(recent_authors)
-
- # ── Risk classification ──
- days = health.get('last_commit_days_ago')
- recent = health['recent_contributors']
- share = health['top_contributor_share']
-
- if health['archived']:
- health['risk'] = 'critical'
- elif health.get('error'):
- health['risk'] = 'unknown'
- elif days and days > 730: # 2+ years no commits
- health['risk'] = 'critical'
- elif days and days > 365: # 1-2 years
- health['risk'] = 'high'
- elif recent == 0:
- health['risk'] = 'high'
- elif recent == 1 and share > 80:
- health['risk'] = 'high'
- elif recent <= 2:
- health['risk'] = 'medium'
- elif share > 80:
- health['risk'] = 'medium'
- else:
- health['risk'] = 'low'
-
- except Exception as e:
- health['error'] = str(e)[:100]
-
- return action_full, health
-
- results = await asyncio.gather(*[check_action(a) for a in unique_actions])
- for action_full, health in results:
- action_health[action_full] = health
-
- print(f" Checked {len(action_health)} action repos", flush=True)
-
- # ── Count by risk level ──
- risk_counts = defaultdict(int)
- for h in action_health.values():
- risk_counts[h['risk']] += 1
-
- print(f" Risk: critical={risk_counts['critical']} high={risk_counts['high']} "
- f"medium={risk_counts['medium']} low={risk_counts['low']}", flush=True)
+ url = f"https://api.github.com/repos/{github_owner}/{repo_name}/contents/{manifest_file}"
+ r = await http_client.get(url, headers=github_headers)
+ if r.status_code == 200:
+ data = r.json()
+ download_url = data.get('download_url')
+ if download_url:
+ content_resp = await http_client.get(download_url, follow_redirects=True)
+ if content_resp.status_code == 200:
+ return manifest_file, content_resp.text
+ except Exception:
+ pass
+ return manifest_file, None
+
+ for idx, repo_name in enumerate(target_repos):
+ if (idx + 1) % 50 == 0:
+ print(f" [{idx+1}/{len(target_repos)}] Fetching manifests...", flush=True)
+
+ # Check which manifest files exist from the tree
+ extras = extras_data.get(f"__extras__:{repo_name}", {})
+
+ # Fetch all manifest files concurrently
+ tasks = []
+ for lang, spec in MANIFEST_SPECS.items():
+ for manifest_file in spec['files']:
+ tasks.append(fetch_manifest(repo_name, manifest_file))
+
+ results = await asyncio.gather(*tasks)
+
+ repo_dep_map = defaultdict(set)
+ for manifest_file, content in results:
+ if content and manifest_file in PARSERS:
+ deps = PARSERS[manifest_file](content)
+ if deps:
+ # Find ecosystem for this manifest
+ for lang, spec in MANIFEST_SPECS.items():
+ if manifest_file in spec['files']:
+ eco = spec['ecosystem']
+ repo_dep_map[eco].update(deps)
+ all_deps[eco].update(deps)
+ break
+
+ if repo_dep_map:
+ repo_deps[repo_name] = {eco: sorted(deps) for eco, deps in repo_dep_map.items()}
+
+ print(f" Found dependencies in {len(repo_deps)} repos", flush=True)
+ for eco, deps in sorted(all_deps.items()):
+ print(f" {eco}: {len(deps)} unique packages", flush=True)
+
+ # ── Check maintainer data for unique dependencies ──
+ dep_health = {} # (ecosystem, pkg) -> health dict
+ dep_users = defaultdict(set) # (ecosystem, pkg) -> set of repos
+
+ # Build reverse map
+ for repo, eco_deps in repo_deps.items():
+ for eco, deps in eco_deps.items():
+ for dep in deps:
+ dep_users[(eco, dep)].add(repo)
+
+ # Only check deps used by 2+ repos, or all if total is manageable
+ total_unique = sum(len(d) for d in all_deps.values())
+ if total_unique > 500:
+ # Focus on shared deps and PyPI/npm/crates (where we have API access)
+ check_deps = [(eco, dep) for (eco, dep), repos in dep_users.items()
+ if len(repos) >= 2 and eco in REGISTRY_CHECKS]
+ print(f" Checking {len(check_deps)} shared dependencies (of {total_unique} total)", flush=True)
+ else:
+ check_deps = [(eco, dep) for eco, dep in dep_users.keys()
+ if eco in REGISTRY_CHECKS]
+ print(f" Checking {len(check_deps)} dependencies", flush=True)
+
+ check_sem = asyncio.Semaphore(5) # Lower concurrency for registry APIs
+
+ async def check_dep(eco, pkg):
+ async with check_sem:
+ checker = REGISTRY_CHECKS.get(eco)
+ if not checker:
+ return (eco, pkg), None
+ result = await checker(pkg)
+ return (eco, pkg), result
+
+ results = await asyncio.gather(*[check_dep(eco, pkg) for eco, pkg in check_deps])
+ for (eco, pkg), health in results:
+ if health:
+ dep_health[(eco, pkg)] = health
+
+ print(f" Got maintainer data for {len(dep_health)} packages", flush=True)
+
+ # ── Classify risk ──
+ risky = [] # (ecosystem, pkg, health, repos)
+ for (eco, pkg), health in sorted(dep_health.items()):
+ mc = health.get('maintainer_count', 0)
+ repos = dep_users.get((eco, pkg), set())
+ risk = 'low'
+ if mc == 0:
+ risk = 'critical'
+ elif mc == 1:
+ risk = 'high'
+ elif mc == 2:
+ risk = 'medium'
+ health['risk'] = risk
+ health['repo_count'] = len(repos)
+ if risk in ('critical', 'high', 'medium'):
+ risky.append((eco, pkg, health, repos))
+
+ risky.sort(key=lambda x: ({'critical': 0, 'high': 1, 'medium': 2}.get(x[2]['risk'], 3),
+ -x[2]['repo_count']))
# ── Generate report ──
lines = []
- lines.append(f"# Bernies: CI Supply Chain Health — {github_owner}\n")
- lines.append("Identifies GitHub Actions dependencies with low bus factor: "
- "few active maintainers, abandoned repos, or single-contributor dominance. "
- "Named after the [Weekend at Bernies](https://github.com/andrew/weekend-at-bernies) "
- "pattern.\n")
+ lines.append(f"# Bernies: Dependency Health — {github_owner}\n")
+ lines.append("Identifies project dependencies with low bus factor: packages maintained "
+ "by too few people. Named after the [Weekend at Bernies]"
+ "(https://github.com/andrew/weekend-at-bernies) pattern where a project "
+ "appears alive but its maintainers have moved on.\n")
lines.append("## Summary\n")
- lines.append(f"**{len(unique_actions)}** unique third-party actions across the org "
- f"(excludes actions/\\*, github/\\*, apache/\\*).\n")
- lines.append("| Risk | Actions | Description |")
- lines.append("|------|---------|-------------|")
- lines.append(f"| CRITICAL | {risk_counts['critical']} | Archived, deleted, or 2+ years without commits |")
- lines.append(f"| HIGH | {risk_counts['high']} | 1-2 years stale, zero recent contributors, or single-maintainer with >80% share |")
- lines.append(f"| MEDIUM | {risk_counts['medium']} | 1-2 active contributors or dominant single contributor |")
- lines.append(f"| LOW | {risk_counts['low']} | 3+ recent contributors, healthy activity |")
+ lines.append(f"- **{len(repo_deps)}** repos with parseable dependency manifests")
+ lines.append(f"- **{total_unique}** unique dependencies across all ecosystems")
+ lines.append(f"- **{len(dep_health)}** checked against registries")
+ lines.append(f"- **{len(risky)}** flagged with maintainer risk\n")
+
+ if not risky:
+ lines.append("No dependencies flagged with maintainer risk.\n")
+ else:
+ lines.append("## Flagged Dependencies\n")
+ lines.append("| Risk | Package | Ecosystem | Maintainers | Used By |")
+ lines.append("|------|---------|-----------|-------------|---------|")
+ for eco, pkg, health, repos in risky:
+ mc = health.get('maintainer_count', '?')
+ names = ', '.join(health.get('maintainers', [])[:3])
+ lines.append(f"| {health['risk'].upper()} | `{pkg}` | {eco} | {mc} ({names}) | {len(repos)} repos |")
+ lines.append("")
+
+ # Detail by risk level
+ for risk_level in ('critical', 'high', 'medium'):
+ items = [(e, p, h, r) for e, p, h, r in risky if h['risk'] == risk_level]
+ if not items:
+ continue
+ lines.append(f"## {risk_level.upper()} ({len(items)} packages)\n")
+ for eco, pkg, health, repos in items:
+ mc = health.get('maintainer_count', '?')
+ version = health.get('version', '?')
+ names = ', '.join(health.get('maintainers', []))
+ source = health.get('source_repo', '')
+
+ lines.append(f"### `{pkg}` ({eco})\n")
+ lines.append(f"- Version: {version}")
+ lines.append(f"- Maintainers: {mc}")
+ if names:
+ lines.append(f"- Names: {names}")
+ if source:
+ lines.append(f"- Source: {source}")
+ lines.append(f"- Used by **{len(repos)} repos**:")
+ for repo in sorted(repos)[:15]:
+ lines.append(f" - {github_owner}/{repo}")
+ if len(repos) > 15:
+ lines.append(f" - ...and {len(repos) - 15} more")
+ lines.append("")
+
+ # Ecosystem breakdown
+ lines.append("## Dependencies by Ecosystem\n")
+ lines.append("| Ecosystem | Repos | Unique Deps | Checked | Flagged |")
+ lines.append("|-----------|-------|-------------|---------|---------|")
+ for eco in sorted(all_deps.keys()):
+ n_repos = len([r for r, d in repo_deps.items() if eco in d])
+ n_deps = len(all_deps[eco])
+ n_checked = len([1 for (e, _), h in dep_health.items() if e == eco])
+ n_flagged = len([1 for e, _, h, _ in risky if e == eco])
+ lines.append(f"| {eco} | {n_repos} | {n_deps} | {n_checked} | {n_flagged} |")
lines.append("")
- # Sort by risk level then blast radius
- RISK_ORDER = {'critical': 0, 'high': 1, 'medium': 2, 'low': 3, 'unknown': 4}
-
- def sort_key(action):
- h = action_health[action]
- return (RISK_ORDER.get(h['risk'], 9), -len(action_users[action]))
-
- # ── Detail by risk level ──
- for risk_level, risk_label in [('critical', 'CRITICAL'), ('high', 'HIGH'),
- ('medium', 'MEDIUM'), ('low', 'LOW')]:
- risky = [a for a in unique_actions if action_health[a]['risk'] == risk_level]
- if not risky:
- continue
-
- lines.append(f"## {risk_label} ({len(risky)} actions)\n")
-
- for action in sorted(risky, key=sort_key):
- h = action_health[action]
- users = action_users[action]
- versions = action_versions[action]
- n_repos = len(set(r for r, _ in users))
-
- lines.append(f"### [{action}](https://github.com/{action})\n")
-
- # Health stats
- stats = []
- if h['archived']:
- stats.append("**ARCHIVED**")
- if h.get('error') == 'repo not found':
- stats.append("**DELETED**")
- if h['last_commit']:
- days = h.get('last_commit_days_ago', '?')
- stats.append(f"Last commit: {h['last_commit']} ({days} days ago)")
- if h['recent_contributors'] is not None:
- stats.append(f"Active contributors (past year): {h['recent_contributors']}")
- if h['total_contributors']:
- stats.append(f"Total contributors: {h['total_contributors']}")
- if h['top_contributor'] and h['top_contributor_share']:
- stats.append(f"Top contributor: {h['top_contributor']} ({h['top_contributor_share']}% of commits)")
-
- for s in stats:
- lines.append(f"- {s}")
-
- # Versions in use
- pinned = [v for v in versions if re.match(r'^[a-f0-9]{40}$', v)]
- unpinned = [v for v in versions if not re.match(r'^[a-f0-9]{40}$', v)]
- if unpinned:
- lines.append(f"- Versions: {', '.join(sorted(unpinned))}")
- if pinned:
- lines.append(f"- SHA-pinned: {len(pinned)} ref(s)")
-
- # Blast radius
- lines.append(f"- **Used by {n_repos} repos** in {len(users)} workflows")
- if n_repos <= 10:
- for repo, wf in sorted(users):
- lines.append(f" - {github_owner}/{repo} `{wf}`")
-
- # Recommendation
- if risk_level == 'critical':
- lines.append(f"- **Action: replace immediately.** Find an alternative or fork.")
- elif risk_level == 'high':
- lines.append(f"- **Action: plan replacement.** Monitor for signs of abandonment.")
-
- lines.append("")
-
lines.append("---\n")
- lines.append(f"*{len(unique_actions)} actions checked. "
- f"{risk_counts['critical']} critical, {risk_counts['high']} high, "
- f"{risk_counts['medium']} medium, {risk_counts['low']} low risk.*")
+ lines.append(f"*Checked {len(dep_health)} dependencies across {len(repo_deps)} repos. "
+ f"{len(risky)} flagged.*")
report = "\n".join(lines)
- # Store in CouchDB
+ # Store
bernies_ns = data_store.use_namespace(f"ci-bernies:{github_owner}")
bernies_ns.set_many({
"latest_report": report,
"latest_data": json.dumps({
'generated_at': datetime.now(timezone.utc).isoformat(),
- 'action_health': action_health,
- 'action_users': {k: list(v) for k, v in action_users.items()},
+ 'dep_health': {f"{e}:{p}": h for (e, p), h in dep_health.items()},
+ 'risky_count': len(risky),
}),
})
- print(f"Report: {len(report)} chars", flush=True)
-
return {"outputText": report}
finally:
diff --git a/gha-review/agents/gha_orchestrator.py b/gha-review/agents/gha_orchestrator.py
index 4ef6c3c..78dd9be 100644
--- a/gha-review/agents/gha_orchestrator.py
+++ b/gha-review/agents/gha_orchestrator.py
@@ -30,34 +30,118 @@ async def run(input_dict, tools):
pushed = []
errors = []
+ collected_files = {} # filepath -> content (for batch commit)
+
+ def collect_file(filename, content):
+ """Stage a file for batch commit."""
+ filepath = f"{write_directory}/{filename}" if write_directory else filename
+ collected_files[filepath] = content
+ pushed.append((filename, "OK"))
+ print(f" ✓ {filename}", flush=True)
+
+ async def push_all_collected(commit_msg):
+ """Push all collected files in a single Git commit."""
+ if not collected_files:
+ return
+
+ import base64
+
+ headers = {
+ "Authorization": f"Bearer {write_pat}",
+ "Accept": "application/vnd.github+json",
+ }
+ api = "https://api.github.com"
- async def push_file(filename, content, commit_msg):
try:
- filepath = f"{write_directory}/{filename}" if write_directory else filename
- result = await gofannon_client.call(
- agent_name="asvs_push_github",
- input_dict={
- "inputText": json.dumps({
- "repo": write_repo,
- "token": write_pat,
- "filePath": filepath,
- }),
- "commitMessage": commit_msg,
- "fileContents": content,
- }
- )
- output = result.get("outputText", "")
- if "error" in output.lower() and "message" in output.lower():
- pushed.append((filename, "ERROR"))
- errors.append(f"Push {filename}: {output[:200]}")
- print(f" ✗ {filename}", flush=True)
- else:
- pushed.append((filename, "OK"))
- print(f" ✓ {filename}", flush=True)
+ # 1. Get current HEAD
+ ref_resp = await http_client.get(
+ f"{api}/repos/{write_repo}/git/ref/heads/main",
+ headers=headers)
+ if ref_resp.status_code != 200:
+ # Try master branch
+ ref_resp = await http_client.get(
+ f"{api}/repos/{write_repo}/git/ref/heads/master",
+ headers=headers)
+ if ref_resp.status_code != 200:
+ errors.append(f"Batch push: could not get branch ref: {ref_resp.status_code}")
+ print(f" ✗ Batch push failed: {ref_resp.status_code}", flush=True)
+ return
+ ref_data = ref_resp.json()
+ head_sha = ref_data["object"]["sha"]
+ branch_ref = ref_data["ref"]
+
+ # 2. Get current tree
+ commit_resp = await http_client.get(
+ f"{api}/repos/{write_repo}/git/commits/{head_sha}",
+ headers=headers)
+ if commit_resp.status_code != 200:
+ errors.append(f"Batch push: could not get commit: {commit_resp.status_code}")
+ return
+ base_tree = commit_resp.json()["tree"]["sha"]
+
+ # 3. Create blobs for each file
+ tree_items = []
+ for filepath, content in sorted(collected_files.items()):
+ blob_resp = await http_client.post(
+ f"{api}/repos/{write_repo}/git/blobs",
+ headers=headers,
+ json={
+ "content": base64.b64encode(content.encode("utf-8")).decode("utf-8"),
+ "encoding": "base64",
+ })
+ if blob_resp.status_code != 201:
+ errors.append(f"Batch push: blob failed for {filepath}: {blob_resp.status_code}")
+ continue
+ blob_sha = blob_resp.json()["sha"]
+ tree_items.append({
+ "path": filepath,
+ "mode": "100644",
+ "type": "blob",
+ "sha": blob_sha,
+ })
+
+ if not tree_items:
+ errors.append("Batch push: no blobs created")
+ return
+
+ # 4. Create tree
+ tree_resp = await http_client.post(
+ f"{api}/repos/{write_repo}/git/trees",
+ headers=headers,
+ json={"base_tree": base_tree, "tree": tree_items})
+ if tree_resp.status_code != 201:
+ errors.append(f"Batch push: tree creation failed: {tree_resp.status_code}")
+ return
+ new_tree_sha = tree_resp.json()["sha"]
+
+ # 5. Create commit
+ commit_create_resp = await http_client.post(
+ f"{api}/repos/{write_repo}/git/commits",
+ headers=headers,
+ json={
+ "message": commit_msg,
+ "tree": new_tree_sha,
+ "parents": [head_sha],
+ })
+ if commit_create_resp.status_code != 201:
+ errors.append(f"Batch push: commit creation failed: {commit_create_resp.status_code}")
+ return
+ new_commit_sha = commit_create_resp.json()["sha"]
+
+ # 6. Update branch ref
+ ref_update_resp = await http_client.patch(
+ f"{api}/repos/{write_repo}/git/{branch_ref}",
+ headers=headers,
+ json={"sha": new_commit_sha})
+ if ref_update_resp.status_code != 200:
+ errors.append(f"Batch push: ref update failed: {ref_update_resp.status_code}")
+ return
+
+ print(f"\n Batch commit: {len(tree_items)} files in 1 commit ({new_commit_sha[:8]})", flush=True)
+
except Exception as e:
- pushed.append((filename, "ERROR"))
- errors.append(f"Push {filename}: {e}")
- print(f" ✗ {filename}: {e}", flush=True)
+ errors.append(f"Batch push: {type(e).__name__}: {e}")
+ print(f" ✗ Batch push error: {e}", flush=True)
async def run_agent(name, **kwargs):
try:
@@ -95,13 +179,11 @@ async def run(input_dict, tools):
else:
# Top-level file, use as-is
target = fname
- await push_file(target, content,
- f"CI scan: {fname} ({github_owner})")
+ collect_file(target, content)
return
except (json.JSONDecodeError, AttributeError):
pass
- await push_file(fallback_name, output,
- f"CI scan: {fallback_name} ({github_owner})")
+ collect_file(fallback_name, output)
def generate_readme(owner):
return f"""# GitHub Actions Security Review: {owner}
@@ -119,7 +201,8 @@ Automated CI/CD security and publishing analysis. Start with [brief.md](brief.md
| [publishing_detail.md](publishing_detail.md) | Enriched: secrets, action SHAs, triggers, run history |
| [publishing_risks.md](publishing_risks.md) | Dangerous publishing patterns |
| [artifact_verification.md](artifact_verification.md) | Registry cross-reference: verified vs phantom packages |
-| [bernies.md](bernies.md) | CI supply chain health: action dependency bus factor |
+| [action_health.md](action_health.md) | GitHub Actions dependency health |
+| [bernies.md](bernies.md) | Project dependency bus factor analysis |
## Per-Channel Reports
@@ -177,6 +260,7 @@ jq -r '.repos[] | select(.trusted_publishing.migration_needed) | [.repo, (.trust
f"ci-security:{github_owner}",
f"ci-publishing-detail:{github_owner}",
f"ci-artifact-verify:{github_owner}",
+ f"ci-action-health:{github_owner}",
f"ci-bernies:{github_owner}",
f"ci-combined:{github_owner}",
]
@@ -227,8 +311,7 @@ jq -r '.repos[] | select(.trusted_publishing.migration_needed) | [.repo, (.trust
publishing_report = await run_agent("gha_publishing",
github_owner=github_owner)
if publishing_report:
- await push_file("publishing.md", publishing_report,
- f"CI scan: publishing ({github_owner})")
+ collect_file("publishing.md", publishing_report)
print("▸ Security analysis...", flush=True)
security_report = await run_agent("gha_security",
@@ -246,40 +329,46 @@ jq -r '.repos[] | select(.trusted_publishing.migration_needed) | [.repo, (.trust
github_owner=github_owner)
await push_multi_file_output(verify_output, "artifact_verification.md")
- print("▸ Bernies (action dependency health)...", flush=True)
+ print("▸ Action health...", flush=True)
+ action_health_report = await run_agent("gha_action_health",
+ github_owner=github_owner,
+ read_pat=read_pat)
+ if action_health_report:
+ collect_file("action_health.md", action_health_report)
+
+ print("▸ Bernies (dependency health)...", flush=True)
bernies_report = await run_agent("gha_bernies",
github_owner=github_owner,
read_pat=read_pat)
if bernies_report:
- await push_file("bernies.md", bernies_report,
- f"CI scan: bernies ({github_owner})")
+ collect_file("bernies.md", bernies_report)
print("▸ Combined review...", flush=True)
review_report = await run_agent("gha_review",
github_owner=github_owner)
if review_report:
- await push_file("review.md", review_report,
- f"CI scan: review ({github_owner})")
+ collect_file("review.md", review_report)
print("▸ Executive brief...", flush=True)
brief_report = await run_agent("gha_brief",
github_owner=github_owner)
if brief_report:
- await push_file("brief.md", brief_report,
- f"CI scan: brief ({github_owner})")
+ collect_file("brief.md", brief_report)
print("▸ JSON export...", flush=True)
json_report = await run_agent("gha_json_export",
github_owner=github_owner)
if json_report:
- await push_file("json_export.json", json_report,
- f"CI scan: JSON export ({github_owner})")
+ collect_file("json_export.json", json_report)
# ── README ──
print("▸ Report index...", flush=True)
readme = generate_readme(github_owner)
- await push_file("README.md", readme,
- f"CI scan: report index ({github_owner})")
+ collect_file("README.md", readme)
+
+ # ── Batch push all files in one commit ──
+ print(f"\n▸ Pushing {len(collected_files)} files...", flush=True)
+ await push_all_collected(f"CI scan: {github_owner} ({len(collected_files)} reports)")
# ══════════════════════════════════════════════════
# Summary
diff --git a/gha-review/agents/gha_publishing.py b/gha-review/agents/gha_publishing.py
index 6569640..a20aa4f 100644
--- a/gha-review/agents/gha_publishing.py
+++ b/gha-review/agents/gha_publishing.py
@@ -50,7 +50,7 @@ async def run(input_dict, tools):
(f" (redacting {redacted_severity})" if redacted_severity else ""), flush=True)
provider = "bedrock"
- model = "us.anthropic.claude-sonnet-4.6"
+ model = "us.anthropic.claude-sonnet-4-6"
configured_params = {"temperature": 0, "reasoning_effort": "disable", "max_tokens": 2048}
classification_cache = data_store.use_namespace(f"ci-classification:{github_owner}")
@@ -384,6 +384,15 @@ async def run(input_dict, tools):
print(f"Found {len(repo_names)} repos in workflow cache\n", flush=True)
+ # Diagnostic: check classification cache health
+ cls_keys = classification_cache.list_keys()
+ cls_meta_count = sum(1 for k in cls_keys if k.startswith("__meta__:"))
+ print(f" Classification cache: {len(cls_keys)} keys, {cls_meta_count} meta entries", flush=True)
+ if cls_meta_count > 0:
+ sample_key = next(k for k in cls_keys if k.startswith("__meta__:"))
+ sample_val = classification_cache.get(sample_key)
+ print(f" Sample: {sample_key} = {str(sample_val)[:100]}", flush=True)
+
# ===== STEP 2: Fetch workflows and classify (parallel) =====
all_results = {}
stats = {"repos_scanned": 0, "repos_with_workflows": 0, "total_workflows": 0,
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org