You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by gs...@apache.org on 2026/10/21 15:52:14 UTC
(tooling-llmao) branch main updated: docs: box-protocol.md replaces vllm-fleet-design.md, as fact not proposal
This is an automated email from the ASF dual-hosted git repository.
gstein pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-llmao.git
The following commit(s) were added to refs/heads/main by this push:
new 328749b docs: box-protocol.md replaces vllm-fleet-design.md, as fact not proposal
328749b is described below
commit 328749b56906e959f2a44061ce6fa4f83e9e868c
Author: Greg Stein <gs...@gmail.com>
AuthorDate: Wed Oct 21 10:37:44 2026 -0500
docs: box-protocol.md replaces vllm-fleet-design.md, as fact not proposal
The September design for the Vast fleet had become the only description
of the box contract, still framed as a proposal with goals, non-goals,
open points and a status line, and its proposal parts had rotted: listen
ports 8000-8002 (the set is 8001-8003), the bearer "from litellm_params"
(it is the HMAC), RunPod marked planned, a config revision described as
if built, and pending assignments wanting a Postgres table.
The new file states what is: the one request a box makes, what it must
present and how the host is resolved, the JSON it gets back field by
field, hosts and listen versus public ports, the three config lifetimes,
the fleet key and the derived vLLM bearer with the accepted leaked-map
risk, provisioning in one paragraph with the runbooks linked, and the
security notes. Goals, non-goals, future extensions and open points are
dropped; the one open item, a retired host's record, has no home and is
not carried.
Co-Authored-By: Claude Fable 5.1 <no...@anthropic.com>
---
README.md | 4 +-
docs/ARCHITECTURE.md | 2 +-
docs/box-protocol.md | 203 ++++++++++++++++++++++++
docs/fleet-reconcile-design.md | 5 +-
docs/vllm-fleet-design.md | 344 -----------------------------------------
hosting/README.md | 3 +-
6 files changed, 211 insertions(+), 350 deletions(-)
diff --git a/README.md b/README.md
index c75a46b..c28dc81 100644
--- a/README.md
+++ b/README.md
@@ -15,7 +15,7 @@ here.
| **This README** | How to run the software |
| **[Help](https://llm.apache.org/help)** (`docs/help/`) | How to use the gateway: keys, connecting tools |
| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | Repo structure |
-| [`docs/vllm-fleet-design.md`](docs/vllm-fleet-design.md) | GPU fleet: control-plane contract, box provisioning |
+| [`docs/box-protocol.md`](docs/box-protocol.md) | How a GPU box asks for its configuration: the fleet key, the vLLM bearer, the JSON |
| [`docs/fleet-reconcile-design.md`](docs/fleet-reconcile-design.md) | Fleet state kept in step with LiteLLM (working document) |
| Infra `p6/modules/llmao` | Production deploy |
@@ -165,7 +165,7 @@ and port. `GET /vllm/config` still comes from `fleet.hosts` (listen ports)
plus `models.yaml`. Boxes fetch it with template `FLEET_KEY`.
See [`hosting/README.md`](hosting/README.md),
-[`docs/vllm-fleet-design.md`](docs/vllm-fleet-design.md) and
+[`docs/box-protocol.md`](docs/box-protocol.md) and
[`docs/fleet-reconcile-design.md`](docs/fleet-reconcile-design.md). Operational detail — what is
running where, and the exact launch commands — is kept out of this repo.
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 43a6842..752acf8 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -67,7 +67,7 @@ keys are team-scoped exceptions (who may create them is an **open RAI
policy** question — see design + `docs/STATUS.md`). Secrets shown once;
metadata in LiteLLM.
-**GPU fleet (control plane):** `docs/vllm-fleet-design.md`. `APP.fleet`
+**GPU fleet (control plane):** `docs/box-protocol.md`. `APP.fleet`
(`llmao/fleet.py`) is built at startup. `GET /vllm/config` (Bearer
`fleet.key`) maps `X-LLMAO-Host` (Vast `PUBLIC_IPADDR`) or
`X-Forwarded-For` / peer to `fleet.hosts`.
diff --git a/docs/box-protocol.md b/docs/box-protocol.md
new file mode 100644
index 0000000..54956b7
--- /dev/null
+++ b/docs/box-protocol.md
@@ -0,0 +1,203 @@
+# How a GPU box asks for its configuration
+
+The contract between a GPU box and llmao: the one request a box makes at
+boot, what it must present, and what it gets back. This is how the fleet
+works today, not a proposal. Box boot on each provider is
+[`../hosting/README.md`](../hosting/README.md); how the running fleet is
+kept in step with LiteLLM is
+[`fleet-reconcile-design.md`](fleet-reconcile-design.md).
+
+```mermaid
+flowchart LR
+ BOX["GPU box<br/>(Vast or RunPod)"]
+ LLMAO["llmao<br/>fleet.hosts + models.yaml"]
+ LITELLM["LiteLLM proxy"]
+ BOX -- "GET /vllm/config<br/>Bearer fleet key<br/>X-LLMAO-Host" --> LLMAO
+ LLMAO -- "{host, servers[]}" --> BOX
+ BOX -- "vllm serve :8001 :8002 :8003" --> LITELLM
+ LLMAO -- "/model/new when Healthy" --> LITELLM
+```
+
+llmao is the source of truth for which models exist, how each is served
+(`models.yaml`), which box serves what on which port (`fleet.hosts`), and
+the bearer LiteLLM presents to each vLLM. The box learns all of it from one
+JSON response. There is no on-disk `servers.yaml` and no long-lived process
+manager on the box: the installer writes Supervisor units and exits.
+
+## The request
+
+```
+GET /vllm/config
+Authorization: Bearer <fleet.key>
+X-LLMAO-Host: <the box's name in fleet.hosts>
+```
+
+- **The fleet key** is the gate. `config.yaml` `fleet.key` is one shared
+ secret, baked into every box template. Missing: 403. Wrong, by a
+ constant-time compare: 403. Not configured on the server: 503.
+- **The host** is the `fleet.hosts` key the answer is built for. When
+ `X-LLMAO-Host` is present it is the only key: no fallback to the peer,
+ and 404 if it is unknown. Vast instances reach llmao through a transparent
+ proxy, so the TCP peer is not the instance and there is no
+ `X-Forwarded-For`. Without the header, the leftmost `X-Forwarded-For`
+ address is used, else the peer. The header is a claim, not a proof; the
+ fleet key is what is checked.
+- **Which name a box sends** is the provider's business, set in that
+ provider's `provision.sh` as `LLMAO_HOST`: the public IP on Vast
+ (`PUBLIC_IPADDR`), the pod id on RunPod (`RUNPOD_POD_ID`). `install_set.py`
+ knows no provider's names.
+- **An unknown host** gets 404, and the fetch is remembered in memory (the
+ host, first and last seen, a count), so the `/models` attention strip can
+ show "fleet key presented from a host not in `fleet.hosts`" and an admin
+ can approve it with its models. Every fetch, known or not, is logged with
+ the claimed host, the peer and `X-Forwarded-For`.
+- **A known host's fetch is noted.** The time of the last config fetch is
+ the anchor for the Loading window: a box that fetched its config and is
+ not yet answering is Loading until `fleet.health_grace_s` has passed, then
+ Stalled.
+
+Boxes fetch `https://llm.apache.org/vllm/config` with default TLS
+verification.
+
+## The response
+
+```json
+{
+ "host": "203.0.113.10",
+ "servers": [
+ {
+ "name": "gemma4-26b",
+ "model": "google/gemma-4-26b-it",
+ "served_model_name": "gemma4-26b",
+ "host": "203.0.113.10",
+ "port": 8001,
+ "api_key": "sk-vllm-...",
+ "args": ["--reasoning-parser", "gemma"],
+ "max_model_len": 32768,
+ "vram_gb": 18,
+ "disk_gb": 60
+ }
+ ]
+}
+```
+
+One entry per vLLM process on the box, joined from the host's `fleet.hosts`
+rows and the `models.yaml` recipe each names.
+
+- `name`: the process name, the row's third element when it has one, else
+ the model name. Supervisor's program name and the log file name.
+- `model`: the HF weights id (`model_info.vllm.model` in `models.yaml`).
+ `served_model_name` is the name vLLM advertises on `/v1/models`.
+- `port`: the **listen** port, what `vllm serve --port` binds. Never the
+ public port a provider maps it to; the box does not need to know that.
+- `api_key`: the bearer this vLLM must require (`--api-key`), derived as
+ below. LiteLLM presents the same value.
+- `args`: extra `vllm serve` arguments. A list of strings, or one string
+ the installer splits.
+- `gpu_memory_utilization`, `max_model_len`, `vram_gb`, `disk_gb`,
+ `kv_bytes_per_token`: present only when the recipe sets them.
+ `gpu_memory_utilization` is normally absent; the box divides the card's
+ memory between its servers from `vram_gb`. `vram_gb` and `disk_gb` are
+ what the box checks before pulling weights.
+
+## Hosts and ports
+
+A `fleet.hosts` entry is the box's name (above) mapped to rows of
+`[model_name, port]` or `[model_name, port, name]`. The port is the
+container listen port, one of 8001, 8002 or 8003 (`VLLM_LISTEN_PORTS`;
+a box gets at most three), plus the handful of older pairs in
+`LEGACY_SERVERS`. The optional name lets two processes share one
+`models.yaml` row.
+
+A provider publishes each listen port on some other public port. llmao
+learns that mapping from the provider's listing (Vast today, RunPod's
+`portMappings` as early-work item 8 of the reconcile design) and uses it
+for everything that dials the box: the LiteLLM deployment's `api_base`,
+the health probe, the `/v1/models` read and the metrics sampler. A fourth
+element on a host row, a public port written by hand, is on its way out
+with that item. Without a GPU provider key configured, public equals
+listen.
+
+llmao owns placement. `models.yaml` says how to serve a model, not where.
+Moving a model between boxes is a config change and a restart of llmao;
+the box templates are identical and never change.
+
+### Three config lifetimes
+
+| lifetime | what | to change it |
+|---|---|---|
+| **baked** | `FLEET_KEY`, `ASFQUART_URL` in the box template | re-provision |
+| **boot** | the assignment: model, port, launch args, `max_model_len` | the box must re-fetch and restart vLLM |
+| **live** | deployments, keys, budgets in LiteLLM | immediate |
+
+Changing `max_model_len` in `models.yaml` does nothing on a box until that
+box restarts vLLM, and nothing compares what llmao intends against what a
+box is running. A config revision reported back by the box was designed
+for that and is not built.
+
+## The fleet key and the vLLM bearer
+
+Two secrets, with different jobs.
+
+**The fleet key** (`fleet.key`) is the one long-lived secret on a box. One
+value for every box and every provider: that is what keeps the templates
+identical and placement a control-plane matter. It gates `GET
+/vllm/config` and nothing else. It does not mint a vLLM bearer.
+
+**The vLLM bearer** is derived, not stored:
+
+```
+HMAC-SHA256(fleet.vllm_api_salt, "llmao.vllm.api_key\n" + host + "\n" + listen_port)
+→ "sk-vllm-" + urlsafe-base64, no padding
+```
+
+`llmao/vllm_api_key.py` is the derivation; `bin/llmao-vllm-api-key`
+computes one by hand. The box needs it before `vllm serve` and LiteLLM
+needs the same value at `/model/new` after the box is Healthy, so it has to
+be computable on both sides from the host and listen port alone. llmao
+keeps no key table and never reads the bearer back from LiteLLM, which
+stores it encrypted. The salt lives only in llmao's `config.yaml` (Puppet
+eyaml), never in a box template, and is not rotated in place: a new salt
+means a new bearer on every box, a vLLM restart, and a new `/model/new`
+each. Instance env injection was tried on Vast and does not work, which is
+why the JSON is how a box learns `--api-key`. The pairs in
+`LEGACY_SERVERS` predate the salt and still run the shared
+`fleet.selfhost_api_key`; `fleet.working_api_key` picks the right one.
+
+**Who can mint.** The salt never leaves llmao, so a third party cannot
+compute a bearer, including a box that copied `FLEET_KEY`. Host and listen
+port are still required inputs.
+
+**Leaked host and port map.** A list of fleet hosts and listen ports can
+leak (config, logs, the public Vast mapping). Accepted for today: the map
+alone does not mint bearers, and membership is not hidden beyond the fleet
+key on `GET /vllm/config`. Revisit if a leaked map should be insufficient
+even for someone who also holds `FLEET_KEY`: that caller can claim
+`X-LLMAO-Host` for each listed host and receive its derived key. Those
+fetches are logged. Per-instance bootstrap tokens stay out of scope until
+that revisit.
+
+## Provisioning, in one paragraph
+
+Both providers run the same installer, `hosting/install_set.py`, fetched
+from this repository at boot. Vast runs it from a stock vLLM container via
+`PROVISIONING_SCRIPT`; RunPod runs it from our own image, whose entrypoint
+is its `provision.sh`. The installer sets `LLMAO_HOST`, fetches the JSON,
+checks `vram_gb` and `disk_gb` against the box, divides the card's memory
+between the servers, writes one Supervisor program per server, and hands
+over to supervisord, which runs `vllm serve` with the listen port, the
+bearer and the arguments from the JSON. Restarts are Supervisor's. The
+runbooks are [`../hosting/vast/README.md`](../hosting/vast/README.md) and
+[`../hosting/runpod/README.md`](../hosting/runpod/README.md).
+
+## Security
+
+- The fleet key is the only long-lived secret a box holds. The vLLM bearer
+ is derived from a salt the box never sees.
+- `GET /vllm/config` is served over HTTPS only.
+- vLLM's `/health` takes no key. A passing probe says something answers at
+ that address, not that it is ours; `/v1/models` with the bearer is the
+ check that it is (the reconcile design's Foreign verdict).
+- Hardening that is possible without changing the rest: instance binding,
+ short-lived bootstrap tokens, a signed token in place of the static fleet
+ key. None is planned until the leaked-map revisit above.
diff --git a/docs/fleet-reconcile-design.md b/docs/fleet-reconcile-design.md
index 46df43b..8b8b5d4 100644
--- a/docs/fleet-reconcile-design.md
+++ b/docs/fleet-reconcile-design.md
@@ -4,8 +4,9 @@ Status: a working document. It is edited as the work lands, so what it
says is the current intent and the current state, not a record of either.
Sections marked **Done** name the commit. The document is meant to become
obsolete: once the reconcile loop is the code, its outcome is folded into
-the architecture documentation (`ARCHITECTURE.md`, and the fleet sections
-of `vllm-fleet-design.md`) and this file is removed.
+the architecture documentation (`ARCHITECTURE.md`) and this file is
+removed. The box side of the fleet, the one request a box makes and what
+it gets back, is `box-protocol.md` and is not changed by this work.
## The problem
diff --git a/docs/vllm-fleet-design.md b/docs/vllm-fleet-design.md
deleted file mode 100644
index 3cf662c..0000000
--- a/docs/vllm-fleet-design.md
+++ /dev/null
@@ -1,344 +0,0 @@
-# Design: Multi-vLLM Fleet on Vast.ai with asfquart Control Plane
-
-**Box contract and control-plane protocol** (host = IP, fleet key,
-`GET /vllm/config` shape, listen vs public, identical templates). How the
-fleet's live state is kept in step with LiteLLM:
-[`fleet-reconcile-design.md`](fleet-reconcile-design.md) (a working
-document; its outcome lands in `ARCHITECTURE.md`). The HMAC `api_key`
-derivation: `llmao/vllm_api_key.py`. Box boot:
-[`../hosting/README.md`](../hosting/README.md).
-
-**Status:** Operating — boxes fetch `GET /vllm/config` (`X-LLMAO-Host` on
-Vast). Listen vs public ports; Vast HostPort (`vast_client`); health-gated
-`/model/new` / `/model/delete`; commercial register at llmao startup.
-Remaining: long vLLM boot, config revision, box smoke. Per-vLLM `api_key`
-is HMAC of `fleet.vllm_api_salt` and is returned on `/vllm/config`
-(§3.1). A leaked host/port map is accepted for today.
-**Date:** 2026-09-10
-**Scope:** One or more Vast.ai GPU instances, each running 1–3 vLLM servers for distinct models, fronted by a LiteLLM proxy managed by an asfquart application.
-
----
-
-## 1. Goals
-
-- Run multiple distinct LLM models efficiently across a small fleet of GPU boxes (initially Vast.ai; later RunPod and others).
-- Amortize fixed instance cost by placing 1–3 vLLM servers on the same machine when VRAM allows.
-- Keep all model secrets and configuration in one place (the asfquart control plane).
-- Use a single shared authentication secret ("fleet key") so GPU boxes can securely fetch their configuration.
-- Support non-trivial per-model vLLM arguments without complicating the GPU-side template.
-- Make the same mechanism reusable for other GPU providers.
-
-**Non-goals (for now):**
-- Automatic scaling / serverless.
-- Per-instance *random* secrets or short-lived tokens. Per-vLLM `api_key` is
- **derived** (HMAC of `fleet.vllm_api_salt`, not `fleet.key`); see
- §3.1. A leaked host/port map is accepted for today.
-- Reverse proxy / path-based routing on the GPU box (LiteLLM talks directly to the vLLM ports).
-
----
-
-## 2. High-Level Architecture
-
-```mermaid
-flowchart TB
- A["asfquart application<br/>(LiteLLM config + secrets + hosts)"]
- B["GET /vllm/config<br/>Authorization: Bearer <fleet-key><br/>host = client IP"]
- C["JSON response<br/>(with secrets)"]
-
- subgraph BoxA["Vast.ai box A — IP in fleet membership"]
- A1["vllm :8000 (model)"]
- A2["vllm :8001 (model)"]
- A3["vllm :8002 (model)"]
- end
-
- subgraph BoxB["Vast.ai box B — IP in fleet membership"]
- B1["vllm :8000 (model)"]
- B2["vllm :8001 (model)"]
- end
-
- Dots["… up to ~6 boxes"]
-
- L["LiteLLM proxy<br/>(OpenAI-compatible)"]
-
- A -- B --> C
- C --> BoxA
- C --> BoxB
- BoxA -- OpenAI-compatible --> L
- BoxB -- OpenAI-compatible --> L
-```
-
-- **asfquart** is the single source of truth for:
- - Which models exist
- - Their LiteLLM configuration
- - The real API keys used between LiteLLM and each vLLM server
- - Logical **hosts** (models + ports on one GPU box, keyed by public IP)
-- Each GPU box receives the **fleet key** on the template. asfquart looks up
- `X-LLMAO-Host` when present (Vast `PUBLIC_IPADDR`; TCP peer may be a
- transparent proxy), else leftmost `X-Forwarded-For`, else
- `request.remote_addr`.
-- At box-start, the provider installer fetches JSON for that IP
- and installs native process units (Vast: Supervisor). There is no on-disk
- `servers.yaml` and no Python process manager.
-
----
-
-## 3. Core Concepts
-
-**Definitions** — `models.yaml`: how to serve each model. **Model** — one
-row (`model_name`, e.g. `gemma4-26b`). **Deployment** — one LiteLLM backend
-row (`/model/new`; `hosted_vllm/` for self-host). **VllmServer** — one vLLM
-process on a host (`port`). Box JSON `servers[].model` is the **HF
-weights id** (`model_info.vllm.model`); that field name is deferred.
-
-### 3.1 Fleet Key
-
-- A single shared secret known to asfquart and to every GPU box.
-- Presented as `Authorization: Bearer <fleet-key>` when fetching configuration.
-- Vast stock container + custom template: `install_set.py` also sends
- `X-LLMAO-Host: $PUBLIC_IPADDR` (no query string). That header is a claim,
- not a proof; the fleet key is the gate.
-- Chosen over per-instance *random* secrets for operational simplicity (one
- value to manage, works across providers). Each vLLM `--api-key` is HMAC of
- `fleet.vllm_api_salt` plus host and listen port (`llmao/vllm_api_key.py`
- is the derivation; `bin/llmao-vllm-api-key` computes one by hand).
- `fleet.key` does not mint that bearer. `/vllm/config` returns it in
- `servers[].api_key`; instance env injection was tried on Vast and does not
- work, so the JSON is how a box learns `--api-key`. The salt lives only in
- llmao `config.yaml` (Puppet eyaml), never in the box template, and is not
- rotated in place: a change means a new bearer on every box, a vLLM restart,
- and a new `/model/new`.
-- **Who can mint.** The salt never leaves llmao, so a third party cannot
- compute a bearer, including a box that copied `FLEET_KEY`. Host and listen
- port are still required inputs.
-- **Leaked host/port map.** A list of fleet IPs and listen ports can leak
- (config, logs, the public Vast mapping). Accepted for today: the map alone
- does not mint bearers, and membership is not hidden beyond the fleet key on
- `GET /vllm/config`. Revisit if a leaked map should be insufficient even for
- someone who also holds `FLEET_KEY`: that caller can still claim
- `X-LLMAO-Host` for each listed IP and receive the derived key. Those
- fetches are logged (claimed host, TCP peer, `X-Forwarded-For`).
- Per-instance bootstrap tokens stay out of scope until that revisit.
-- Acceptable risk for a small, operator-controlled fleet. Can be hardened later (instance binding, short-lived tokens, etc.) without changing the rest of the design.
-
-### 3.2 Hosts
-
-A **host** is a GPU box public IP. Its value is a list of `[model, port]` or
-`[model, port, name]` rows. That **port is the container listen port**
-(`vllm serve --port` / box JSON). Vast's proxy publishes a different public
-HostPort; llmao records that as `VllmServer.public_port`.
-Without `providers.vast`, public equals listen. Optional **name** lets two
-processes share a models.yaml row (e.g. two qwen3 on one box).
-
-asfquart owns placement. `models.yaml` is how to serve, not where. Changing
-placement is a control-plane change only; GPU templates stay identical (shared
-`FLEET_KEY`).
-
-llmao config is the source of truth; LiteLLM's Postgres is the working copy
-the proxy needs in order to route. A deployment's `litellm_params.api_base`
-is the host and public port, and its `model_info` carries `llmao_key` (the
-host and LISTEN port, the deployment's identity), `asf_api_base`,
-`self_hosted` and `access_groups`. Registration is health-gated: a
-deployment is added when its vLLM is Healthy and deleted when it is
-Unhealthy or Stalled. How that is kept in step is
-[`fleet-reconcile-design.md`](fleet-reconcile-design.md).
-
-### 3.3 Config lifetimes
-
-Config has three change costs, and conflating them causes most of the confusion
-about why an edit "did nothing":
-
-| lifetime | what | to change it |
-|---|---|---|
-| **baked** | `ASFQUART_URL`, `FLEET_KEY` in the template | re-provision |
-| **boot** | the vLLM assignment — model, port, launch args | box re-fetches, restarts vLLM |
-| **live** | deployments, keys, budgets in LiteLLM | immediate |
-
-Changing `max_model_len` does nothing until that box restarts vLLM. A revision
-hash on the config response, reported back by the box, is what makes the
-difference between intended and running visible.
-
-### 3.4 Host config JSON
-
-asfquart builds JSON from `hosts.<client-ip>` joined to `models.yaml`. Box
-boot is `hosting/README.md` (Vast `PROVISIONING_SCRIPT` → `provision.sh` →
-`install_set.py`). Same payload; never a `servers.yaml`.
-
----
-
-## 4. Endpoint Contract
-
-```
-GET /vllm/config
-Authorization: Bearer <fleet-key>
-X-LLMAO-Host: <public-ip> # required on Vast; absent locally is peer/XFF
-
-Response: 200 application/json
-```
-
-- asfquart validates the fleet key (not OAuth).
-- If `X-LLMAO-Host` is present, that is the only `fleet.hosts` key (404 if
- unknown). Do not fall back to the proxy peer.
-- Else: `X-Forwarded-For` or `request.remote_addr`.
-- Emits JSON (host, servers: name, HF weights id, host, port, args, API keys).
-
----
-
-## 5. JSON schema
-
-```json
-{
- "host": "127.0.0.1",
- "servers": [
- {
- "name": "model-a",
- "model": "org/model-name",
- "host": "127.0.0.1",
- "port": 8000,
- "api_key": "sk-...",
- "gpu_memory_utilization": 0.42,
- "max_model_len": 16384,
- "vram_gb": 9,
- "disk_gb": 12,
- "args": ["--dtype", "auto"]
- }
- ]
-}
-```
-
-Catalog model (`model_info.vllm`): HF weights id, optional util, max len,
-`vram_gb`, `disk_gb`, `args`. All but the weights id and `args` are omitted
-from the JSON when the catalog does not set them.
-Host row (`fleet_path` → `hosts`): `[model_name, port]` or `[model_name, port, name]`.
-`api_key` comes from `litellm_params` (same secret LiteLLM presents).
-
-Notes:
-- `args` may be a list of strings or a single string; the installer normalises either form.
-- `gpu_memory_utilization` is normally absent. The box then works out a
- fraction for each server from the memory it finds and the servers'
- `vram_gb`; a value sent here is used as given. See `hosting/README.md`.
-- `vram_gb` and `disk_gb` are what the box checks before pulling weights.
-- Boxes fetch `https://llm.apache.org/vllm/config` with default TLS verify.
-
----
-
-## 6. GPU Box Provisioning Flow
-
-1. Instance is created from a common Vast.ai template with at least:
-
- ```bash
- -e FLEET_KEY=<shared-secret>
- ```
-
- plus the usual image, ports (8000–8002), disk size, SSH, etc.
-
-2. Provisioning (`hosting/vast/provision.sh` → `install_set.py`):
- - GETs JSON from asfquart.
- - Checks the servers' `vram_gb` and `disk_gb` against the box and stops
- if they do not fit.
- - Divides the card's memory between the servers.
- - Writes one Supervisor `[program:vllm-<name>]` per server.
- - `supervisorctl update`. supervisord runs:
-
- ```bash
- vllm serve <model> --host 0.0.0.0 --port <port> \
- --api-key <api_key> \
- --gpu-memory-utilization … \
- --max-model-len … \
- <extra args…>
- ```
-
- Logs go to `$DATA_DIRECTORY/logs/<name>.log` on Vast. Restart is Supervisor
- `autorestart` / `startretries`.
-
-3. LiteLLM (managed by asfquart) is configured with matching `api_base` values and the same API keys, so it can reach each vLLM server.
-
----
-
-## 7. Vast.ai Template Requirements (summary)
-
-- Image: `vastai/vllm:…` or `vllm/vllm-openai:…` (or a thin derivative).
-- Launch mode: SSH (or Jupyter + SSH).
-- Ports: 8000, 8001, 8002 mapped.
-- Disk: large enough for the heaviest set of models that will be assigned.
-- Environment:
- - `FLEET_KEY` (template)
- - `ASFQUART_URL`
-- On-create: `provision.sh` → `install_set.py` (JSON → Supervisor units).
-
-The template is identical for every box. Placement is keyed by public IP in
-`fleet_path` → `hosts`.
-
-### RunPod (planned)
-
-- Image: our `llmao-vllm-box` (registry TBD / Infra). Not a stock RunPod
- Jupyter template. COPY the installer; do not curl GitHub at boot.
-- Same env: `FLEET_KEY`, `ASFQUART_URL=https://llm.apache.org`,
- `DATA_DIRECTORY`.
-- Supervisord, same units as Vast (several `vllm serve` per pod).
-- Ports: expose every listen port. Public mapping via REST v2
- `portMappings` or the HTTP proxy `https://<podId>-<listen>.proxy.runpod.net`
- once proven. The optional fourth `fleet.hosts` field is a stopgap.
-- See `hosting/runpod/README.md`.
-
----
-
-## 8. Security Considerations
-
-- Fleet key is the only long-lived secret that must be present on GPU boxes.
-- Real model API keys exist only inside asfquart and in the short-lived YAML that is fetched at boot.
-- Endpoint must be served over HTTPS.
-- Recommended: restrict endpoint reachability (private network, Tailscale, Cloudflare Access, IP allow-list, etc.).
-- vLLM `GET /health` has **no API key**. The host:port from fleet membership
- must be on a private path (Tailscale / WireGuard / allow-list). Tunnels are
- out of scope for v1.
-- Future hardening options (not required for v1):
- - Instance-ID binding
- - Short-lived bootstrap tokens
- - Signed JWTs instead of a static fleet key
-
----
-
-## 9. Future Extensions
-
-- Same endpoint + fleet key used by RunPod (or any other provider) instances.
-- Additional set metadata (preferred GPU type, minimum VRAM, etc.) for scheduling.
-- Automatic re-fetch of configuration on SIGHUP or periodic interval.
-- Do **not** use LiteLLM `GET /health` as the vLLM boot probe (it runs real
- completions). asfquart probes vLLM `/health` and, rarely, LiteLLM `/health`
- only to detect **skew**. Health-gated `/model/new` is implemented.
-
----
-
-## 10. Open Points / Decisions Still Soft
-
-- Restart policy details (Supervisor `autorestart` / give up after N, …).
-- Where pending assignments live between "host added" and "vLLM healthy".
- A route only exists once the server is serving, so the assignment needs a
- home before that. Preferred: a small table in the same Postgres.
-- Whether a retired host keeps a record. Deleting a route deletes the row.
-- Remaining Vast operational nits (framework works; boxes fetch config).
-
-**Resolved:** `GET /vllm/config` (no path) is keyed from `fleet.hosts`.
-That stays; it is not derived from LiteLLM rows. A fleet-key fetch from an
-IP not in `fleet.hosts` is remembered in memory and shown on the `/models`
-attention strip, where an admin can approve it with its models; Add into
-LiteLLM is on the existing row and waits until that vLLM is serving
-(`models-dashboard-design.md`). Deployments live in Postgres
-(`store_model_in_db` in YAML and/or env). Registration is health-gated.
-`ASFQUART_URL` + template `FLEET_KEY`; no `VLLM_SET`; no on-disk
-`servers.yaml`; no Werkzeug ProxyFix on Quart ASGI.
-
----
-
-## 11. Leftover implementation
-
-1. Smoke remaining box issues.
-2. Vast public HostPort — **done** (`vast_client` when `providers.vast.api_key` is set).
-3. `/model/new` / `/model/delete` — **done**. Delete id via `model_info.asf_api_base`
- (`models.yaml` params; LiteLLM encrypts `litellm_params` on readback).
-4. Config revision on `/vllm/config`, reported back by the box, so a stale
- vLLM cannot pretend to be current (see 3.3).
-
----
-
-*End of design document.*
diff --git a/hosting/README.md b/hosting/README.md
index 4c6a8b2..4d39b4e 100644
--- a/hosting/README.md
+++ b/hosting/README.md
@@ -3,7 +3,8 @@
Artifacts that run **on** a GPU instance (Vast.ai and RunPod). They are
not part of the asfquart process.
-**Design (why):** [`docs/vllm-fleet-design.md`](../docs/vllm-fleet-design.md).
+**The contract (what a box asks for and gets):**
+[`docs/box-protocol.md`](../docs/box-protocol.md).
**Control plane (all providers):** asfquart serves `GET /vllm/config`
as JSON from `config.yaml` `fleet.hosts` plus `models.yaml`. Bearer fleet
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org