You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by gs...@apache.org on 2026/09/20 10:30:23 UTC
(tooling-llmao) branch main updated: docs: drop leftover catalog wording
This is an automated email from the ASF dual-hosted git repository.
gstein pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-llmao.git
The following commit(s) were added to refs/heads/main by this push:
new fd111ba docs: drop leftover catalog wording
fd111ba is described below
commit fd111ba81d2afb2dd4770211a5c1172a7ee62146
Author: Greg Stein <gs...@gmail.com>
AuthorDate: Sun Sep 20 05:29:17 2026 -0500
docs: drop leftover catalog wording
Say models.yaml / definitions for the admin file, and LiteLLM
deployment for a backend row. Route stays an HTTP path in this app.
Co-authored-by: Grok <gr...@x.ai>
---
docs/STATUS.md | 2 +-
docs/fleet-state.md | 22 +++++++++++-----------
docs/vllm-fleet-design.md | 14 +++++++-------
hosting/runpod/README.md | 4 ++--
hosting/vast/install_set.py | 4 ++--
templates/fleet.ezt | 2 +-
tests/test_fleet.py | 10 +++++-----
tests/test_fleet_deployments.py | 2 +-
8 files changed, 30 insertions(+), 30 deletions(-)
diff --git a/docs/STATUS.md b/docs/STATUS.md
index a7f30b8..b50f536 100644
--- a/docs/STATUS.md
+++ b/docs/STATUS.md
@@ -43,7 +43,7 @@ overhead and run roughly half these. Single-stream decode is
memory-bandwidth bound, so the models sit closer together than their sizes
suggest — the difference between them is concurrency, not per-request speed.
-**Fit validation:** the catalog declares `vram_gb` and `disk_gb`, and a box checks them against `nvidia-smi` and `statvfs` before pulling weights. Requirements sum across co-resident servers. Silent when `nvidia-smi` is absent — an unknown is not a failure.
+**Fit validation:** `models.yaml` declares `vram_gb` and `disk_gb`, and a box checks them against `nvidia-smi` and `statvfs` before pulling weights. Requirements sum across co-resident servers. Silent when `nvidia-smi` is absent — an unknown is not a failure.
**Observed state:** `/fleet` shows measured KV cache against the served context window, scraped from vLLM's `/metrics` on the transition into SERVING. A `--max-model-len` above what the cache holds makes vLLM hang rather than error, which was previously only visible by reading a startup log.
diff --git a/docs/fleet-state.md b/docs/fleet-state.md
index 489dd2e..8642b73 100644
--- a/docs/fleet-state.md
+++ b/docs/fleet-state.md
@@ -25,7 +25,7 @@ State lives in LiteLLM's database. There is no second store.
| field | fleet meaning |
|---|---|
-| `model_name` | the catalog model, and the routing key |
+| `model_name` | the `models.yaml` id (LiteLLM model group) |
| `litellm_params.api_base` | **which host, which port** |
| `litellm_params.api_key` | the bearer token for that vLLM |
| `model_info` | arbitrary dict — carries the recipe and provenance |
@@ -37,8 +37,8 @@ caller's IP, return their `model_info.vllm` blocks and ports.
Set `general_settings.store_model_in_db: true` in `litellm.yaml`. Puppet may
also set `STORE_MODEL_IN_DB=True` in the process env. Without the flag,
-`/model/new` returns HTTP 500. The YAML catalog is the recipe, not the live
-route table.
+`/model/new` returns HTTP 500. `models.yaml` is the recipe, not the live
+deployment table.
---
@@ -112,10 +112,10 @@ the confusion:
|---|---|---|
| **baked** | `ASFQUART_URL`, `FLEET_KEY` in the instance template | re-provision |
| **boot** | the vLLM assignment — model, port, launch args | box must re-fetch and restart vLLM |
-| **live** | routes, keys, budgets in LiteLLM | immediate |
+| **live** | deployments, keys, budgets in LiteLLM | immediate |
-Changing `max_model_len` does nothing until that box restarts vLLM. The catalog
-says one thing and the server does another, silently.
+Changing `max_model_len` does nothing until that box restarts vLLM. The
+definitions file says one thing and the server does another, silently.
**This is a second kind of skew.** `check_config_skew` compares llmao against
LiteLLM. Nothing compares llmao's *intended* launch args against what a box is
@@ -155,11 +155,11 @@ Same `model_name`, different recipes — fine:
Different `model_name` required:
-- different served context window — a 32k route cannot take a 100k prompt
+- different served context window — a 32k deployment cannot take a 100k prompt
- different reasoning parser, or thinking on by default versus off
- anything that changes the shape of a response
-**If two routes share a `model_name`, their caller-visible parameters must
+**If two deployments share a `model_name`, their caller-visible parameters must
match.** Otherwise identical requests behave differently depending on which
backend they land on. Where a pool is uneven, advertise the **minimum** — a
pool with a 40k and a 128k server advertises 40k, or it is not a pool.
@@ -183,7 +183,7 @@ model_info:
Three reasons to separate `source` from `model`: the box can check fit before
downloading; credentials differ by source kind; and a box that already holds
-the weights should not re-download because the catalog names an HF repo.
+the weights should not re-download because `models.yaml` names an HF repo.
### 4.3 Host capacity is discovered, not declared
@@ -227,7 +227,7 @@ which makes "nobody else could hold it" an assumption.
- **Group by provider in the UI.** The useful question is "how many rented
boxes are still alive", not an alphabetical list of IPs.
- **`model_name` pools need a minimum-advertising rule** (§4.1), which cannot
- be a static catalog value once a pool is uneven.
+ be a static `models.yaml` value once a pool is uneven.
- **A `notes` field is worth considering.** "Rented for the superset scan, kill
after" is obvious for a week and unrecoverable after a month.
@@ -244,7 +244,7 @@ which makes "nobody else could hold it" an assumption.
Health state is correctly ephemeral — a live measurement, not a record.
-With routes in Postgres, a lost database means the proxy no longer knows what
+With deployments in Postgres, a lost database means the proxy no longer knows what
to proxy, not merely who may call it. Virtual keys are stored hashed and shown
once at mint, so recovery also means re-minting every key and reconfiguring
every consumer.
diff --git a/docs/vllm-fleet-design.md b/docs/vllm-fleet-design.md
index d062c7c..f315dcb 100644
--- a/docs/vllm-fleet-design.md
+++ b/docs/vllm-fleet-design.md
@@ -97,11 +97,11 @@ HostPort; llmao records that as `VllmServer.public_port`.
Without `fleet.vast`, public equals listen. Optional **name** lets two
processes share a models.yaml row (e.g. two qwen3 on one box).
-asfquart owns placement. The catalog is how to serve, not where. Changing
+asfquart owns placement. `models.yaml` is how to serve, not where. Changing
placement is a control-plane change only; GPU templates stay identical (shared
`FLEET_KEY`).
-Fleet state lives in LiteLLM: a route's `api_base` is the host and port, and
+Fleet state lives in LiteLLM: a deployment's `api_base` is the host and port, and
`model_info` carries the recipe and provenance. See
[fleet-state.md](fleet-state.md) for ownership, lifecycle, and recovery.
@@ -114,7 +114,7 @@ about why an edit "did nothing":
|---|---|---|
| **baked** | `ASFQUART_URL`, `FLEET_KEY` in the template | re-provision |
| **boot** | the vLLM assignment — model, port, launch args | box re-fetches, restarts vLLM |
-| **live** | routes, keys, budgets in LiteLLM | immediate |
+| **live** | deployments, keys, budgets in LiteLLM | immediate |
Changing `max_model_len` does nothing until that box restarts vLLM. A revision
hash on the config response, reported back by the box, is what makes the
@@ -122,9 +122,9 @@ difference between intended and running visible.
### 3.4 Host config JSON
-asfquart builds JSON from `hosts.<client-ip>` joined to the catalog. Vast
-`install_set.py` fetches `GET /vllm/config` at box-start and writes Supervisor
-programs. Same payload; never a `servers.yaml`.
+asfquart builds JSON from `hosts.<client-ip>` joined to `models.yaml`. Box
+boot is `hosting/README.md` (Vast `PROVISIONING_SCRIPT` → `provision.sh` →
+`install_set.py`). Same payload; never a `servers.yaml`.
---
@@ -283,7 +283,7 @@ The template is identical for every box. Placement is keyed by public IP in
1. Smoke remaining box issues.
2. Vast public HostPort — **done** (`vast_client` when `fleet.vast.api_key` is set).
3. `/model/new` / `/model/delete` — **done**. Delete id via `model_info.asf_api_base`
- (catalog params; LiteLLM encrypts `litellm_params` on readback).
+ (`models.yaml` params; LiteLLM encrypts `litellm_params` on readback).
4. Config revision on `/vllm/config`, reported back by the box, so a stale
vLLM cannot pretend to be current (see 3.3).
diff --git a/hosting/runpod/README.md b/hosting/runpod/README.md
index 11749fa..eadc6aa 100644
--- a/hosting/runpod/README.md
+++ b/hosting/runpod/README.md
@@ -15,12 +15,12 @@ image and `fleet.hosts` row must satisfy.
- Template env: `FLEET_KEY`, `ASFQUART_URL=https://llm.apache.org`,
`DATA_DIRECTORY` (typically `/workspace`). TLS verify is on; there is
no skip flag.
-- **Supervisord** runs one `vllm serve` per catalog row (several listens
+- **Supervisord** runs one `vllm serve` per `models.yaml` row (several listens
in one pod). That is the usual shape for a GPU *pod*, not a one-process
web container, and it matches Vast.
- Weights and logs live on the volume (`$DATA_DIRECTORY/hf-cache`,
`.../logs`), not in the image.
-- Pre-pull fit check: catalog `vram_gb` / `disk_gb` against the card and
+- Pre-pull fit check: `models.yaml` `vram_gb` / `disk_gb` against the card and
`statvfs`. That is **not** vLLM: vLLM only sizes KV after the weights
are on disk (`torch` / HIP `mem_get_info`). NVIDIA probe today is
`nvidia-smi`; missing probe is not a failure. AMD (`rocm-smi`) when we
diff --git a/hosting/vast/install_set.py b/hosting/vast/install_set.py
index 6283389..db54d28 100755
--- a/hosting/vast/install_set.py
+++ b/hosting/vast/install_set.py
@@ -139,7 +139,7 @@ def servers_from_config(data: dict[str, Any]) -> list[dict[str, Any]]:
def free_vram_gb() -> float | None:
"""Free VRAM on GPU 0, or None if nvidia-smi is unavailable.
- Capacity is read from the card rather than declared in the catalog: a
+ Capacity is read from the card rather than declared in models.yaml: a
hand-typed figure is wrong the first time a provider supplies a different
GPU than was ordered, and with rented instances that is a matter of when.
"""
@@ -178,7 +178,7 @@ def check_fit(specs: list[dict[str, Any]], *, data_dir: str) -> list[str]:
Checked before writing units because the alternative is a fifteen-minute
weights pull followed by an engine-init failure whose message does not
- mention memory. Requirements come from the catalog; capacity is read from
+ mention memory. Requirements come from models.yaml; capacity is read from
the box.
Silent when nvidia-smi is unavailable or a model declares no vram_gb --
diff --git a/templates/fleet.ezt b/templates/fleet.ezt
index 5919f9e..1200675 100644
--- a/templates/fleet.ezt
+++ b/templates/fleet.ezt
@@ -95,7 +95,7 @@
</table>
</div>
[else]
- <p class="text-muted">No intended deployments (empty fleet.hosts and no commercial catalog rows).</p>
+ <p class="text-muted">No intended deployments (empty fleet.hosts and no commercial models.yaml rows).</p>
[end]
</div>
[include "footer.ezt"]
diff --git a/tests/test_fleet.py b/tests/test_fleet.py
index ee3a456..e25c1e9 100644
--- a/tests/test_fleet.py
+++ b/tests/test_fleet.py
@@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.
-"""config_for_host: JSON for a client IP from fleet.hosts + catalog."""
+"""config_for_host: JSON for a client IP from fleet.hosts + models.yaml."""
from pathlib import Path
@@ -195,10 +195,10 @@ def test_norm_base_makes_skew_comparison_match():
"""The comparison this exists for.
check_config_skew compares VllmServer.api_base against what LiteLLM reports.
- VllmServer.api_base ends in /v1; a route pushed before that, or a catalog
- entry, may lack it. Without normalisation on both sides the same route
- never matches, and the skew check reports a missing route that is actually
- present.
+ VllmServer.api_base ends in /v1; a deployment pushed before that, or a
+ models.yaml row, may lack it. Without normalisation on both sides the
+ same deployment never matches, and the skew check reports a missing
+ deployment that is actually present.
"""
assert _norm_base("http://100.105.28.100:8003/v1") == _norm_base("http://100.105.28.100:8003")
diff --git a/tests/test_fleet_deployments.py b/tests/test_fleet_deployments.py
index 6fe64f0..03eca60 100644
--- a/tests/test_fleet_deployments.py
+++ b/tests/test_fleet_deployments.py
@@ -35,7 +35,7 @@ def test_from_cfg_one_deployment_per_vllm():
assert dep.api_base == "http://127.0.0.1:8001/v1"
-def test_commercial_deployment_from_catalog():
+def test_commercial_deployment_from_definitions():
models = list(load_models(EXAMPLE))
models.append(
EasyDict(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org