You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/09/15 10:05:06 UTC
(magpie) branch main updated: fix(privacy-llm): carve llm.apache.org out of the apache.org default approval (#1226)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new c5955016 fix(privacy-llm): carve llm.apache.org out of the apache.org default approval (#1226)
c5955016 is described below
commit c595501673742102b0cdc51da11273afb07d243b
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Tue Sep 15 16:35:01 2026 +0630
fix(privacy-llm): carve llm.apache.org out of the apache.org default approval (#1226)
The approved-LLM gate default-approved any *.apache.org host for
foundation private data, on the rationale that an Apache domain implies
infra under ASF governance. LLMAO (llm.apache.org) went live in
September 2026 and breaks that assumption: it serves self-hosted models
from rented third-party GPU hardware (Vast.ai, RunPod), and its
operators state pilot traffic must be treated as visible to gateway
admins — public and project-internal material only, no credentials or
embargoed work.
Left as-is, the gate would have waved <private-list> and <security-list>
content, including embargoed CVE detail, onto that path without an
adopter ever declaring it. models.md had already anticipated the case as
a reason the blanket rule might need re-classification.
Add an exact-host carve-out list, checked before the domain rule, and
record the reasoning where adopters will find it. The carve-out binds
the private-data gate only: skills working on public issues, PRs or
releases may use the gateway freely.
Also record the ASF defaults for the gateway — models, context windows,
connection environment variables, and the known limitations that matter
to an adopter, chiefly that tool use over the Anthropic-compatible path
is broken upstream, which is what blocks Magpie's skills from running
against it today.
Source: Andrew Musselman, "llmao progress Sep 14", discuss@rai.apache.org.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_01PQ4eEL4e2QUJMZ8AhfyKTx
---
docs/rfcs/RFC-AI-0004.md | 4 +-
docs/vendor-neutrality.md | 2 +-
organizations/ASF/organization.md | 76 ++++++++++++++++++++++++++
tools/privacy-llm/checker/src/checker/check.py | 22 ++++++++
tools/privacy-llm/checker/tests/test_check.py | 29 ++++++++++
tools/privacy-llm/models.md | 46 ++++++++++++++--
tools/spec-loop/specs/privacy-llm-gate.md | 7 +++
7 files changed, 179 insertions(+), 7 deletions(-)
diff --git a/docs/rfcs/RFC-AI-0004.md b/docs/rfcs/RFC-AI-0004.md
index 65fa2788..001c3c78 100644
--- a/docs/rfcs/RFC-AI-0004.md
+++ b/docs/rfcs/RFC-AI-0004.md
@@ -220,7 +220,7 @@ The reference implementation (see [`docs/setup/secure-agent-internals.md`](http
1. **Pluggable backend.** The framework's skills cite no model name in their flow logic. "Use the strongest model available" / "use a fast model for this lookup step" are acceptable hints; "must be Claude Sonnet 4.5" is not.
2. **Pluggable adapters.** Private-mailing-list, CVE-tool, release-process, and audit-finding ingest MUST live behind adapter modules. The reference implementation's `tools/<adapter>/` directories (`tools/cve-tool-vulnogram/`, `tools/gmail/`, `tools/ponymail/`) are this pattern.
-3. **Pilot diversity.** Validation runs (per the Apache Magpie [MISSION.md](https://github.com/apache/magpie/blob/main/MISSION.md)) cover at least one frontier-model backend, at least one fully-local inference setup (Ollama / vLLM / equivalent), and at least one Apache-hosted or Apache-aligned endpoint. A framework that only validates against one vendor is a vendor-locked framework that has not noticed yet.
+3. **Pilot diversity.** Validation runs (per the Apache Magpie [MISSION.md](https://github.com/apache/magpie/blob/main/MISSION.md)) cover at least one frontier-model backend, at least one fully-local inference setup (Ollama / vLLM / equivalent), and at least one Apache-hosted or Apache-aligned endpoint. A framework that only validates against one vendor is a vendor-locked framework that has not noticed yet. The Apache-hosted leg now has a concrete target: **LLMAO** (`llm.apache.org`), th [...]
4. **Privacy-LLM gating is vendor-neutral by construction.** Private content (security reports, embargoed CVE detail, PMC-private mail) flows only to LLMs the project's PMC has explicitly approved. "Approved" is per-PMC, not per-framework — the framework's contribution is the gate check, not the policy. See [`tools/privacy-llm/`](https://github.com/apache/magpie/blob/main/tools/privacy-llm/) for the reference gate.
5. **License + IP posture.** Framework code AL2.0 / MIT. Skills AL2.0. Generated artefacts (commit messages, PR bodies, advisory drafts) inherit the maintainer's commit licence; the framework MUST NOT introduce a vendor's model-output licence by reference.
@@ -319,7 +319,7 @@ The reference implementation operationalises this principle in [`tools/privacy-
### Five concrete consequences
-1. **Approved-LLM gate.** The project's PMC declares the set of LLMs that may receive private content. The list is per-PMC, not per-framework — Apache Magpie's gate-check tool (`tools/privacy-llm/checker/`) enforces the gate but the *policy* is the project's. The reference list defaults to "Claude Code trusted; `*.apache.org` auto-approved; `localhost` for local-inference setups; everything else requires explicit opt-in."
+1. **Approved-LLM gate.** The project's PMC declares the set of LLMs that may receive private content. The list is per-PMC, not per-framework — Apache Magpie's gate-check tool (`tools/privacy-llm/checker/`) enforces the gate but the *policy* is the project's. The reference list defaults to "Claude Code trusted; `*.apache.org` auto-approved; `localhost` for local-inference setups; everything else requires explicit opt-in." The apache.org default is a **rebuttable** one, not a guarantee: i [...]
2. **Redact-before-read.** When a skill fetches Gmail or PonyMail private content, it pipes the fetched bytes through the framework's PII redactor (`tools/privacy-llm/redactor/`) *before* the LLM sees them. Reporter names → `N-<hash>`, email addresses → `E-<hash>`, IPs → `IP-<hash>`. The skill operates on the hash-prefixed identifiers; the reverse map lives only on the maintainer's local disk (mode 0600, never committed), and the *reveal* step happens at draft-write time inside the maint [...]
3. **Reporter-PII is redacted, but reporter *credit* is not.** The redaction is for in-context PII — the reporter's name and email when the agent is reasoning about routing, triage, deduplication. The reporter's *publicly-credited* identity (e.g., "Reported by Jane Smith" in the CVE record's `credits[]` field) is a deliberate output and passes through unredacted, only after the maintainer confirms the credit shape with the reporter on the inbound thread.
4. **Confidentiality scrub before public emission.** Every skill that emits a public artefact (advisory email, public PR body, public CVE-record JSON, GitHub Security Advisory) MUST run a confidentiality scrub against the draft body: regex match for `CVE-\d{4}-\d{4,7}` (forbidden in pre-disclosure public PRs), reporter names from the private mapping table, mailing-list addresses, and any string the project's policy file enumerates as private. The scrub fires before the draft is shown to [...]
diff --git a/docs/vendor-neutrality.md b/docs/vendor-neutrality.md
index ce6b1af7..9de1443d 100644
--- a/docs/vendor-neutrality.md
+++ b/docs/vendor-neutrality.md
@@ -628,7 +628,7 @@ Harness → substrate tools it supports:
| Default-approved endpoint class | Examples |
|---|---|
| Claude Code itself | The agent invoking the skill |
-| *.apache.org-hosted endpoints | A future ASF-hosted inference endpoint at e.g. `inference.apache.org`; an in-tracker endpoint at `<project>.apache.org/llm/` |
+| *.apache.org-hosted endpoints | An ASF-hosted inference endpoint at e.g. `inference.apache.org`; an in-tracker endpoint at `<project>.apache.org/llm/`. **Not** `llm.apache.org` — see carve-outs |
| Local-only inference | Ollama serving a local model, vLLM on the user's workstation, llama.cpp embedded in a CLI helper |
| Air-gapped on-prem | A PMC-hosted inference appliance on a private VLAN |
diff --git a/organizations/ASF/organization.md b/organizations/ASF/organization.md
index b596d55e..9c5040f4 100644
--- a/organizations/ASF/organization.md
+++ b/organizations/ASF/organization.md
@@ -14,6 +14,7 @@
- [Forwarders](#forwarders)
- [Mail provider](#mail-provider)
- [Archive system](#archive-system)
+ - [Inference endpoint](#inference-endpoint)
- [Project metadata](#project-metadata)
- [Release process](#release-process)
- [Roster](#roster)
@@ -151,6 +152,81 @@ archive_system:
mail_archive_url: https://lists.apache.org
```
+## Inference endpoint
+
+The Foundation-wide sanctioned-inference gateway, **LLMAO**
+(`llm.apache.org`), live since September 2026. A committer authenticates
+with a personal access token; spend is attributed per key. Projects
+inherit this as the ASF-hosted option for the vendor-neutrality
+requirement in
+[RFC-AI-0004 § Principle 3](../../docs/rfcs/RFC-AI-0004.md); it does not
+displace whatever agent harness a maintainer already runs.
+
+```yaml
+inference:
+ gateway: https://llm.apache.org
+ auth: pat # committer personal access token
+ status: pilot # not GA — see privacy_class and limitations
+ # Privacy classification for the approved-LLM gate. LLMAO is NOT
+ # default-approved for foundation private data: it serves from rented
+ # third-party GPU hardware and pilot traffic is visible to llmao admins.
+ # See tools/privacy-llm/models.md — "Carve-outs from the *.apache.org rule".
+ privacy_class: project-internal # public + project-internal only
+ recommended_model: gemma4-26b # reasoning off by default — see note below
+ models:
+ - id: gemma4-26b
+ context_tokens: 131072
+ modalities: [text, vision]
+ tools: true
+ reasoning_on_by_default: false
+ - id: qwen3.8-27b
+ context_tokens: 131072
+ modalities: [text, vision]
+ tools: true
+ reasoning_on_by_default: true
+ - id: qwen3-8b
+ context_tokens: 40960
+ modalities: [text]
+ tools: true
+ reasoning_on_by_default: true
+ known_limitations:
+ # Tool use over the Anthropic-compatible path is broken upstream:
+ # LiteLLM routes it to vLLM's /v1/responses with a tool_choice shape
+ # vLLM rejects. Plain conversation is unaffected. Magpie skills are
+ # tool-driven, so they cannot run against this gateway until it lands.
+ - anthropic_tool_use_broken
+ # spend_usd reports 0.00 for self-hosted models — no cost map yet, so
+ # budget-based routing decisions cannot be made from gateway data.
+ - budgets_do_not_meter_self_hosted
+ # No automatic restart: a host restart leaves the box up, model down.
+ - no_automatic_model_restart
+```
+
+**Pick a model whose reasoning is off by default.** A reasoning model
+emits nothing while it thinks; agent clients abandon the stream and
+retry, so the box runs the same generation twice for a response nobody
+reads. `gemma4-26b` is the default for that reason.
+
+**Connecting an agent.** Claude Code talks to the gateway with
+environment variables alone — LiteLLM exposes `/v1/messages`, so no shim
+is needed:
+
+```bash
+export ANTHROPIC_BASE_URL=https://llm.apache.org
+export ANTHROPIC_AUTH_TOKEN=<your PAT>
+export ANTHROPIC_MODEL=gemma4-26b
+```
+
+Throughput is single-stream and memory-bandwidth bound, so the models sit
+closer together than their parameter counts suggest (~128 tok/s for
+`gemma4-26b`, ~46–54 tok/s for the others). The difference shows up under
+concurrency, where `qwen3.8-27b` reaches 20+ simultaneous requests and
+`gemma4-26b` reaches 4. All published figures come from synthetic load;
+recorded real usage is still only a few hundred requests.
+
+Source: Andrew Musselman, *"llmao progress Sep 14"*,
+`discuss@rai.apache.org`, 2026-09-14.
+
## Project metadata
```yaml
diff --git a/tools/privacy-llm/checker/src/checker/check.py b/tools/privacy-llm/checker/src/checker/check.py
index a5b5f737..1b45e7e9 100644
--- a/tools/privacy-llm/checker/src/checker/check.py
+++ b/tools/privacy-llm/checker/src/checker/check.py
@@ -49,6 +49,20 @@ from checker.config import (
# Hosts that resolve to "local-only inference" — not over the wire.
_LOCAL_HOSTS = frozenset({"localhost", "127.0.0.1", "::1"})
+# ``*.apache.org`` hosts that are explicitly NOT default-approved for
+# foundation private data, despite matching the apache.org rule below.
+# The blanket apache.org approval assumes infra-level ASF governance;
+# these endpoints do not meet it and are carved out by name. See
+# ``tools/privacy-llm/models.md`` — "Carve-outs from the apache.org rule".
+_APACHE_ORG_CARVE_OUTS = {
+ "llm.apache.org": (
+ "LLMAO pilot: serves self-hosted models from rented third-party GPU "
+ "hardware (Vast.ai, RunPod), and its operators state that pilot traffic "
+ "must be treated as visible to llmao admins — public and "
+ "project-internal material only, no credentials or embargoed work"
+ ),
+}
+
# Free-text matches for the always-approved Claude Code entry.
# Case-insensitive, matched as a substring of the bullet's raw
# text. Adopters may write "Claude Code (the agent ...)" or
@@ -76,6 +90,14 @@ def _approve_by_default_rules(entry: LLMEntry) -> Verdict | None:
return Verdict(entry, False, f"unparsable URL host in {entry.url!r}")
if host in _LOCAL_HOSTS:
return Verdict(entry, True, f"local-only inference at {host} (default-approved)")
+ if host in _APACHE_ORG_CARVE_OUTS:
+ return Verdict(
+ entry,
+ False,
+ f"{host} is carved out of the *.apache.org default approval "
+ f"({_APACHE_ORG_CARVE_OUTS[host]}); declare it as an opt-in "
+ f"entry if the PMC accepts the residency terms",
+ )
if host.endswith(".apache.org") or host == "apache.org":
return Verdict(entry, True, f"*.apache.org-hosted endpoint at {host} (default-approved)")
return None
diff --git a/tools/privacy-llm/checker/tests/test_check.py b/tools/privacy-llm/checker/tests/test_check.py
index f225307a..7f23b387 100644
--- a/tools/privacy-llm/checker/tests/test_check.py
+++ b/tools/privacy-llm/checker/tests/test_check.py
@@ -97,6 +97,35 @@ def test_subdomain_apache_org_is_approved():
assert v.approved is True
+def test_llmao_gateway_is_carved_out_of_apache_org_approval():
+ """``llm.apache.org`` matches the apache.org rule but must NOT be approved.
+
+ The LLMAO pilot serves self-hosted models from rented third-party GPU
+ hardware and its operators state pilot traffic is visible to llmao
+ admins, so it fails the infra-governance assumption the blanket
+ apache.org approval rests on.
+ """
+ [v] = check.check_stack(
+ _cfg(
+ [
+ LLMEntry(
+ raw="ASF LLMAO gateway at https://llm.apache.org/",
+ url="https://llm.apache.org/",
+ )
+ ]
+ )
+ )
+ assert v.approved is False
+ assert "carved out" in v.reason
+
+
+def test_llmao_carve_out_does_not_affect_other_apache_hosts():
+ """The carve-out is by exact host, not a prefix or suffix match."""
+ for url in ("https://llm.airflow.apache.org/", "https://inference.apache.org/v1/"):
+ [v] = check.check_stack(_cfg([LLMEntry(raw=f"ASF at {url}", url=url)]))
+ assert v.approved is True, f"expected {url} to stay approved, got {v.reason}"
+
+
def test_apache_org_lookalike_is_not_approved():
"""``apache.org-attacker.example.com`` must NOT match the apache.org rule."""
[v] = check.check_stack(
diff --git a/tools/privacy-llm/models.md b/tools/privacy-llm/models.md
index 3ed4b81f..f469b332 100644
--- a/tools/privacy-llm/models.md
+++ b/tools/privacy-llm/models.md
@@ -7,6 +7,7 @@
- [Approved-LLM registry](#approved-llm-registry)
- [The default-approved entries](#the-default-approved-entries)
+ - [Carve-outs from the `*.apache.org` rule](#carve-outs-from-the-apacheorg-rule)
- [The opt-in entries — adopter declares explicitly](#the-opt-in-entries--adopter-declares-explicitly)
- [The pre-flight check](#the-pre-flight-check)
- [Adopter config — `<project-config>/privacy-llm.md`](#adopter-config--project-configprivacy-llmmd)
@@ -42,7 +43,7 @@ default-approved Claude Code instance and passes).
| Class | Rationale | Examples |
|---|---|---|
| **Claude Code itself** | The Claude-Code instance running framework skills is treated as an approved privacy model for the data it directly processes. See [`docs/setup/privacy-llm.md` — Claude Code trust boundary](../../docs/setup/privacy-llm.md#claude-code-trust-boundary) for the rationale and the limits of this default. | The agent invoking the skill |
-| **`*.apache.org`-hosted endpoints** | Anything served from an Apache Software Foundation domain runs on infra under ASF governance — data residency, retention, and access are bounded by the ASF infra agreement. | A future ASF-hosted inference endpoint at e.g. `inference.apache.org`; an in-tracker endpoint at `<project>.apache.org/llm/` |
+| **`*.apache.org`-hosted endpoints** | Anything served from an Apache Software Foundation domain *and running on infra under ASF governance* — data residency, retention, and access bounded by the ASF infra agreement. An apache.org endpoint that does not meet that assumption is carved out by name below. | An ASF-hosted inference endpoint at e.g. `inference.apache.org`; an in-tracker endpoint at `<project>.apache.org/llm/`. **Not** `llm.apache.org` — see carve-outs |
| **Local-only inference** | The data never leaves the user's machine. No external party (cloud LLM operator, network operator, log aggregator) can observe it. | Ollama serving a local model, vLLM on the user's workstation, llama.cpp embedded in a CLI helper |
| **Air-gapped on-prem** | Same rationale as local inference, scaled to a contributor's organisation. The model server runs on infra the adopter operationally controls and which has no path to a third-party LLM operator. | A PMC-hosted inference appliance on a private VLAN |
@@ -54,6 +55,39 @@ the
[`<project-config>/privacy-llm.md`](#adopter-config--project-configprivacy-llmmd)
declaration shape is what the checker parses.
+## Carve-outs from the `*.apache.org` rule
+
+An apache.org host listed here matches the domain rule above but is
+**not** default-approved for foundation private data, because it does
+not meet the infra-governance assumption that rule rests on. The
+checker rejects these by exact host before the domain rule is applied.
+
+| Host | Why it is carved out | Revisit when |
+|---|---|---|
+| `llm.apache.org` (LLMAO) | Serves self-hosted models from **rented third-party GPU hardware** (Vast.ai, RunPod), not ASF-operated infra. The service's own pilot guidance is that traffic must be treated as *visible to llmao admins* — public and project-internal material is fine, credentials and embargoed work are not. | The pilot ends, the hardware moves under ASF operation, or the ASF publishes a retention-and-access model for it |
+
+**Scope of this carve-out.** It governs the privacy-LLM gate only —
+that is, the question *"may this endpoint receive `<private-list>` or
+`<security-list>` content?"*. It says nothing about using LLMAO for
+public work: a skill operating on public issues, public PRs, or
+published releases may route through `llm.apache.org` freely, because
+that content is not what this gate protects.
+
+What the service currently retains, per its operators: token counts,
+model, latency, and a hashed key for attribution. The columns that
+could hold prompt and completion text exist and are switched off —
+stated explicitly as *a current state rather than a principle*, with
+prompt history flagged as something the ASF will likely want later,
+"a deliberate decision with retention limits and an access model, made
+openly rather than arrived at by default". Both GPU providers are on
+secure tiers (ISO 27001 datacentre partners, signed DPAs, SOC 2 Type
+II), and host-operator inspection of a live request is contractually
+prohibited — but that is a live window on rented hardware, not an
+ASF-governed boundary, which is what the blanket rule assumes.
+
+Source: Andrew Musselman, *"llmao progress Sep 14"*,
+`discuss@rai.apache.org`, 2026-09-14.
+
## The opt-in entries — adopter declares explicitly
Every other LLM endpoint requires the adopter to declare it
@@ -92,6 +126,7 @@ reporter's own identity intact) run this check at Step 0:
"currently configured"
3. For every entry in the stack, decide approved? per:
- Claude Code → ✓ default-approved
+ - Host in the apache.org carve-out list → ✗ (see below)
- URL ending in .apache.org → ✓ default-approved
- Hostname ∈ {localhost, 127.0.0.1, ::1} → ✓ default-approved
- Listed under "approved third-party" with a complete
@@ -198,9 +233,12 @@ the framework adopts until such a policy lands. Specifically:
data, the framework will narrow the default and bump the
registry version.
- The `*.apache.org` blanket approval assumes infra-level
- governance; if a future ASF endpoint runs at `*.apache.org` but
- proxies to a third-party LLM, that endpoint may need
- re-classification.
+ governance; an ASF endpoint at `*.apache.org` that proxies to a
+ third-party LLM, or runs on hardware the ASF rents rather than
+ operates, needs re-classification. This has already happened once:
+ see *Carve-outs from the `*.apache.org` rule* for `llm.apache.org`.
+ Treat the blanket rule as a default that each new ASF inference
+ endpoint must be checked against, not a guarantee.
When ASF Legal does ratify a list, this file becomes the
*pointer* to that list rather than the list itself, and the
diff --git a/tools/spec-loop/specs/privacy-llm-gate.md b/tools/spec-loop/specs/privacy-llm-gate.md
index 55fb7c46..b6766355 100644
--- a/tools/spec-loop/specs/privacy-llm-gate.md
+++ b/tools/spec-loop/specs/privacy-llm-gate.md
@@ -47,6 +47,13 @@ artefact for leakage before emission.
- **Policy is the PMC's; the gate is the framework's.** The approved-LLM
list is per-PMC. The default: agent host trusted, `*.apache.org`
auto-approved, `localhost` for local inference, everything else opt-in.
+- **The `*.apache.org` default is rebuttable.** It assumes the endpoint
+ runs on infra under ASF governance. An apache.org host that does not
+ meet that assumption is carved out by exact host and rejected before
+ the domain rule applies — `llm.apache.org` (LLMAO) is the first, since
+ it serves from rented third-party GPU hardware and its pilot traffic is
+ visible to gateway admins. The carve-out binds the private-data gate
+ only; public-content skills may use such an endpoint freely.
- **Redact before read; reveal locally.** Skills operate on hashed
identifiers; the reverse map never goes to an LLM and is never committed.
- **Reporter credit is preserved** (CVE `credits[]`) only after the