You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by gs...@apache.org on 2026/10/16 05:46:24 UTC

(tooling-llmao) 02/02: hosting/runpod: an image that provisions once, then runs supervisord

This is an automated email from the ASF dual-hosted git repository.

gstein pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-llmao.git

commit 0920aed604b713b257d63ab578e200c291346055
Author: Greg Stein <gs...@gmail.com>
AuthorDate: Fri Oct 16 00:45:49 2026 -0500

    hosting/runpod: an image that provisions once, then runs supervisord
    
    RunPod pods run our image: vllm/vllm-openai:v0.25.1 plus supervisor,
    with runpod/provision.sh as the entrypoint. The installer is not baked
    in. provision.sh fetches the shared install_set.py on first boot, so a
    change to it needs no new image.
    
    A RunPod restart keeps only the volume, so the provisioned state lives
    there: /workspace/.llmao holds the units (conf.d), the installer that
    ran, and a `provisioned` marker written only after the units are in.
    
    - Until provisioned, each boot fetches the installer and runs it with
      --no-reload, retrying with backoff (15s doubling to 5 min). The usual
      first failure is a pod created before its fleet.hosts row; retrying
      means adding the row is enough.
    - Once provisioned, a boot fetches nothing, not GitHub and not
      /vllm/config. To change the set, delete the marker and restart.
    - supervisord.conf, its socket and its pidfile are rewritten on every
      boot off the volume, so a stale socket from an unclean stop cannot
      trip the next start. supervisord then runs as PID 1 and restarts a
      vLLM server that dies without touching the others.
    - LLMAO_HOST comes from RUNPOD_PUBLIC_IP; a pod without a public IP
      refuses to start.
    
    Tests run the real script under bash with curl and supervisord stubbed:
    first boot, a provisioned boot fetching nothing, retries after failed
    fetches and a failing installer, and a pod with no public IP.
    
    Not yet built or run on a pod. runpod/README.md, now a runbook, lists
    what the first pod has to confirm.
    
    Co-Authored-By: Claude Opus 5.5 <no...@anthropic.com>
---
 README.md                      |   2 +-
 hosting/README.md              |  24 ++++--
 hosting/runpod/Dockerfile      |  41 +++++++++
 hosting/runpod/README.md       | 185 ++++++++++++++++++++++++-----------------
 hosting/runpod/provision.sh    | 115 +++++++++++++++++++++++++
 tests/test_runpod_provision.py | 152 +++++++++++++++++++++++++++++++++
 6 files changed, 432 insertions(+), 87 deletions(-)

diff --git a/README.md b/README.md
index 3a70e5e..84f1af1 100644
--- a/README.md
+++ b/README.md
@@ -232,6 +232,6 @@ certs/                   mkcert PEMs + README
 llmao/                   seam, auth, models, fleet, proxy/ (LiteLLM client)
 hosting/install_set.py   GPU-box installer, shared by every provider
 hosting/vast/            provision.sh (Vast on-create)
-hosting/runpod/          RunPod template notes (image not built yet)
+hosting/runpod/          Dockerfile + provision.sh (RunPod entrypoint)
 tests/                   offline seam, fleet, hosting installer
 ```
diff --git a/hosting/README.md b/hosting/README.md
index 52616ad..091f554 100644
--- a/hosting/README.md
+++ b/hosting/README.md
@@ -1,7 +1,7 @@
 # GPU-box hosting
 
-Artifacts that run **on** a GPU instance (Vast.ai now; RunPod next). They
-are not part of the asfquart process.
+Artifacts that run **on** a GPU instance (Vast.ai and RunPod). They are
+not part of the asfquart process.
 
 **Design (why):** [`docs/vllm-fleet-design.md`](../docs/vllm-fleet-design.md).
 
@@ -20,7 +20,9 @@ write Supervisor units; there is no long-lived Python launcher.
 | `install_set.py` | Shared by every provider: GET JSON, write a `vllm-*.conf` per server, `supervisorctl update` (or not, with `--no-reload`) |
 | `vast/provision.sh` | Vast on-create: set `LLMAO_HOST`, curl the installer from GitHub, run it |
 | `vast/README.md` | Vast.ai runbook |
-| `runpod/README.md` | RunPod runbook (image not built yet) |
+| `runpod/Dockerfile` | RunPod image: stock vLLM plus supervisord and `runpod/provision.sh` |
+| `runpod/provision.sh` | RunPod entrypoint: provision once (retrying), then `exec supervisord` |
+| `runpod/README.md` | RunPod runbook |
 
 ## Box environment
 
@@ -32,19 +34,22 @@ Required:
   and logs are `$DATA_DIRECTORY/hf-cache` and `.../logs`, not in the config JSON.
 - `LLMAO_HOST` — the box's public IP, sent as `X-LLMAO-Host`. Each provider's
   `provision.sh` sets it from that provider's own variable (Vast:
-  `PUBLIC_IPADDR`), so `install_set.py` knows no provider's names
+  `PUBLIC_IPADDR`; RunPod: `RUNPOD_PUBLIC_IP`), so `install_set.py` knows
+  no provider's names
 - `PROVISIONING_SCRIPT` — Vast only: raw URL of `provision.sh`
 
 ## Boot
 
 ```text
 Vast:   PROVISIONING_SCRIPT → provision.sh → curl install_set.py → GET JSON → supervisor units → vllm serve
-RunPod: image entrypoint → COPY'd installer → GET JSON → supervisor units → vllm serve
+RunPod: image entrypoint provision.sh → curl install_set.py → GET JSON → units on the volume → exec supervisord → vllm serve
+        (once provisioned, a boot skips straight to exec supervisord)
 ```
 
-Vast still curls `install_set.py` from `refs/heads/main`. Pin that URL to a
-**commit SHA** before treating instance-create as production-safe. RunPod
-pins by **image digest**.
+Both curl `install_set.py` from `refs/heads/main`. Pin that URL to a
+**commit SHA** before treating instance-create as production-safe: a broken
+`main` would stop every new box from provisioning. The RunPod image itself
+is pinned by its tag or digest in the template.
 
 ### Vast `PROVISIONING_SCRIPT` vs `install_set.py`
 
@@ -66,7 +71,8 @@ throwaway instance proves execute is shebang (`./provisioning.sh`) rather
 than `bash /provisioning.sh`. Docs and the saved `.sh` name assume bash.
 `install_set.py` has no shebang today (ASF license first).
 
-RunPod’s image COPY’s the installer; it does not use `provision.sh`.
+RunPod has its own `runpod/provision.sh`, baked into the image as the
+entrypoint; see [`runpod/README.md`](runpod/README.md).
 
 ## Fit check
 
diff --git a/hosting/runpod/Dockerfile b/hosting/runpod/Dockerfile
new file mode 100644
index 0000000..77386ce
--- /dev/null
+++ b/hosting/runpod/Dockerfile
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The RunPod box: stock vLLM, plus supervisord and the boot script.
+#
+# Deliberately thin. The installer is NOT here: provision.sh fetches it on
+# first boot, so a change to it needs no new image. Nor are weights; they
+# go to the volume. This image changes only when provision.sh or the vLLM
+# version does.
+#
+#   docker build -t <registry>/llmao-vllm-box:<tag> hosting/runpod
+
+# A pinned version, not :latest -- every pod created from the template
+# would otherwise get whatever vLLM was newest that day.
+FROM vllm/vllm-openai:v0.25.1
+
+# supervisord is the one thing the stock image lacks: it restarts a vLLM
+# server that dies without disturbing the others on the box.
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends supervisor \
+ && rm -rf /var/lib/apt/lists/*
+
+COPY provision.sh /opt/llmao/provision.sh
+
+# Replaces the stock ENTRYPOINT ["vllm", "serve"]. The template's start
+# command must be left empty: anything there arrives as arguments to this.
+ENTRYPOINT ["/opt/llmao/provision.sh"]
diff --git a/hosting/runpod/README.md b/hosting/runpod/README.md
index 7c18f8e..fcbba01 100644
--- a/hosting/runpod/README.md
+++ b/hosting/runpod/README.md
@@ -1,80 +1,111 @@
-# RunPod runbook (knowledge; image not built yet)
-
-RunPod templates pull a **Docker image**. Vast uses a stock vLLM template
-and curls `install_set.py` at create. Same control plane, different
-artifact.
-
-Nothing in this directory is a running image yet. Registry location is a
-placeholder until Infra says where to push. The notes below are what the
-image and `fleet.hosts` row must satisfy.
-
-## Same as Vast
-
-- `GET /vllm/config` by **client IP**, Bearer `FLEET_KEY`.
-- JSON listen ports only. The box never sees the public mapping.
-- Template env: `FLEET_KEY`, `ASFQUART_URL=https://llm.apache.org`,
-  `DATA_DIRECTORY` (typically `/workspace`). TLS verify is on; there is
-  no skip flag.
-- **Supervisord** runs one `vllm serve` per `models.yaml` row (several listens
-  in one pod). That is the usual shape for a GPU *pod*, not a one-process
-  web container, and it matches Vast.
-- Weights and logs live on the volume (`$DATA_DIRECTORY/hf-cache`,
-  `.../logs`), not in the image.
-- Pre-pull fit check: `models.yaml` `vram_gb` / `disk_gb` against the card and
-  `statvfs`. That is **not** vLLM: vLLM only sizes KV after the weights
-  are on disk (`torch` / HIP `mem_get_info`). NVIDIA probe today is
-  `nvidia-smi`; missing probe is not a failure. AMD (`rocm-smi`) when we
-  have a ROCm image. VRAM sums over the pod's servers; disk counts once per
-  distinct model.
-- The same installer divides the card between the pod's servers; see
-  [`../README.md`](../README.md) (sizing).
-
-## Different from Vast
-
-| | Vast | RunPod |
-|--|------|--------|
-| Image | stock `vastai/vllm` (or similar) | **our** image (`IMAGE_REGISTRY_PLACEHOLDER/llmao-vllm-box`) |
-| Installer | curl `install_set.py` from GitHub **main** | **COPY** in the image (boot must not depend on GitHub) |
-| Pin | later: curl a **commit SHA** | image **digest** |
-| Public port | Vast API HostPort (`vast_client`) | see Ports below |
-
-Pin the Vast curl before create-instance is treated as production-safe: a
-broken `main` would brick every new box. The image digest is the same
-kind of pin for RunPod.
-
-## Ports (open; 4th host-row slot is a stopgap)
-
-`fleet.hosts` already accepts `[model, listen, name|null, public]`. That
-was added so a RunPod box could get an `api_base` at all: RunPod does not
-expose a Vast-style HostPort to us today, and it **reassigns** the TCP
-public port on every pod recreate.
-
-We do **not** want operators editing that fourth number forever. Once
-self-deploy is formalized (create from template, then read the mapping),
-prefer one of:
-
-1. **HTTP proxy (stable URL).** Template exposes `N/http`. Public origin
-   is `https://<podId>-<listen>.proxy.runpod.net`. The listen port is in
-   the hostname, so recreate does not need a new fourth slot. Only use
-   this as `api_base` after we prove LiteLLM and long generations survive
-   that proxy (timeouts, TLS, HTTP/1.1).
-2. **Direct TCP + API.** Template exposes `N/tcp`. Public port changes.
-   RunPod **REST v2** `GET /pods/{id}` → `portMappings` can fill
-   `public_port` the way `vast_client` fills HostPort. GraphQL exists but
-   is deprecated (~early 2027); new code should be REST.
-
-Until one of those is wired, keep the fourth slot. It may go away.
-
-The template must still **expose every listen port** the set uses (max 10
-HTTP ports on RunPod). Control plane JSON stays listen-only.
-
-## Image sketch (next, not this note)
-
-`FROM vllm/vllm-openai:…` (pin a digest later), install supervisord,
-COPY the shared installer, `CMD`/`ENTRYPOINT` that fetches config, writes
-units, `exec supervisord -n`. No baked weights.
-
-## Smoke (once a pod exists)
+# RunPod runbook
+
+RunPod pods run **our** image: stock vLLM plus supervisord and
+[`provision.sh`](provision.sh) as the entrypoint. The installer is the shared
+[`../install_set.py`](../install_set.py), fetched on first boot rather than
+baked in, so changing it needs no new image.
+
+Not yet run on a real pod. What is untested is listed at the end.
+
+## Image
+
+```bash
+docker build -t <registry>/llmao-vllm-box:<tag> hosting/runpod
+docker push <registry>/llmao-vllm-box:<tag>
+```
+
+The registry is still to be settled with Infra. Rebuild only when
+`provision.sh` or the vLLM version in the `Dockerfile` changes. The vLLM
+version is pinned (`v0.25.1`), not `:latest`, so every pod created from the
+template runs the same engine.
+
+## Template
+
+- **Image:** `<registry>/llmao-vllm-box:<tag>`
+- **Start command: empty.** The image's entrypoint is `provision.sh`; anything
+  here arrives as its arguments, which it ignores with a warning.
+- **Volume** mounted at `/workspace`, large enough for the HF cache.
+- **Public IP** required: `provision.sh` refuses to start without
+  `RUNPOD_PUBLIC_IP`, because it is how the box says which host it is.
+- **Expose** every listen port the box's `fleet.hosts` rows use (RunPod
+  allows at most 10 HTTP ports); see Ports below.
+- **Environment:**
+
+  ```bash
+  FLEET_KEY=<shared-secret>
+  ASFQUART_URL=https://llm.apache.org
+  # Optional:
+  # DATA_DIRECTORY=/workspace            (default)
+  # LLMAO_INSTALLER_URL=<raw URL>        pin install_set.py to a commit SHA
+  ```
+
+Add the pod's `RUNPOD_PUBLIC_IP` to `fleet.hosts` in `config.yaml`. Order does
+not matter: a pod created before its row retries until llmao knows it.
+
+## Boot
+
+On every start, [`provision.sh`](provision.sh):
+
+1. If `$DATA_DIRECTORY/.llmao/provisioned` is missing: clears
+   `.llmao/conf.d`, fetches `install_set.py`, and runs it with `--no-reload`
+   to write one Supervisor unit per server into `.llmao/conf.d`. On any
+   failure it retries, waiting 15s and doubling to at most 5 min. It writes
+   `provisioned` only after the units are in place.
+2. Writes `/etc/supervisor/supervisord.conf` (socket and pidfile under
+   `/run/llmao`), including `.llmao/conf.d/*.conf`.
+3. `exec supervisord -n`: supervisord is PID 1 and restarts any `vllm serve`
+   that dies, without touching the others.
+
+A restart keeps only the volume, so everything that must survive is there:
+the units, the marker, the installer that ran, weights (`hf-cache`) and logs.
+Once provisioned, a boot fetches nothing: not GitHub, not `/vllm/config`.
+
+Retrying covers a "will not fit" refusal too. It will not succeed until
+`models.yaml` or `fleet.hosts` changes, and the pod log says why on each try.
+
+## Changing a pod's set
+
+The set is frozen at provisioning. To change it:
+
+```bash
+rm /workspace/.llmao/provisioned
+```
+
+then restart the pod. The next boot clears the old units and provisions
+afresh. Stopping the servers first is not needed: a restart stops them.
+
+## Ports (open)
+
+`fleet.hosts` accepts `[model, listen, name|null, public]`. The fourth slot
+exists because RunPod does not give us a Vast-style HostPort lookup, and it
+**reassigns** the public TCP port on every pod recreate. Operators should not
+have to keep editing that number. Once creating pods is automated, prefer one
+of:
+
+1. **HTTP proxy (stable URL).** Expose `N/http`; the public origin is
+   `https://<podId>-<listen>.proxy.runpod.net`. Recreate does not change it.
+   Use it as `api_base` only after proving LiteLLM and long generations
+   survive that proxy (timeouts, TLS, HTTP/1.1).
+2. **Direct TCP + API.** Expose `N/tcp`; RunPod REST v2 `GET /pods/{id}` →
+   `portMappings` can fill `public_port` the way `vast_client` fills HostPort.
+   GraphQL is deprecated; new code should use REST.
+
+Until one of those is wired, keep the fourth slot.
+
+## Smoke
 
 1. `supervisorctl status`
 2. `curl -sS -H "Authorization: Bearer <api_key>" http://127.0.0.1:<listen>/v1/models`
+3. `$DATA_DIRECTORY/logs/<name>.log`: `Model loading took … GiB` (compare with
+   `vram_gb`). The boot itself logs to the pod's container log, prefixed
+   `llmao-provision:`.
+
+## Untested until the first pod
+
+- The image has not been built. The `Dockerfile` assumes the vLLM image runs
+  as root and has `curl` and `python3`, as its upstream Dockerfile shows.
+- That the IP the pod reaches llm.apache.org **from** matches
+  `RUNPOD_PUBLIC_IP`. llmao looks the box up by `X-LLMAO-Host`, so it need not,
+  but a proxy that strips headers would break the lookup.
+- RunPod's handling of a container whose start command is empty when the
+  image sets its own `ENTRYPOINT`.
diff --git a/hosting/runpod/provision.sh b/hosting/runpod/provision.sh
new file mode 100755
index 0000000..5ab2fcb
--- /dev/null
+++ b/hosting/runpod/provision.sh
@@ -0,0 +1,115 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# RunPod container entrypoint: provision once, then run supervisord as PID 1.
+#
+# A RunPod restart keeps only the volume ($DATA_DIRECTORY, /workspace by
+# default); everything else in the container is reset. So the provisioned
+# state lives on the volume:
+#
+#   $DATA_DIRECTORY/.llmao/install_set.py    the installer that ran
+#   $DATA_DIRECTORY/.llmao/conf.d/*.conf     one Supervisor unit per server
+#   $DATA_DIRECTORY/.llmao/provisioned       written last, once the units are in
+#
+# Until `provisioned` exists, every boot fetches the installer and runs it,
+# retrying until it succeeds. The usual first failure is not GitHub: it is a
+# pod created before its fleet.hosts row, which llmao refuses until the row
+# is added. Once provisioned, a boot fetches nothing -- not the installer,
+# not /vllm/config. To change the set, delete `provisioned` and restart.
+#
+# supervisord's own config, socket and pidfile are NOT on the volume. They
+# are rewritten on every boot, so a stale socket left by an unclean stop
+# cannot trip the next start.
+
+set -euo pipefail
+
+log() { echo "llmao-provision: $*" >&2; }
+
+if [ "$#" -gt 0 ]; then
+  log "ignoring start-command arguments ($*); leave the template's start command empty"
+fi
+
+: "${FLEET_KEY:?FLEET_KEY must be set by the RunPod template}"
+: "${ASFQUART_URL:?ASFQUART_URL must be set by the RunPod template}"
+: "${RUNPOD_PUBLIC_IP:?RUNPOD_PUBLIC_IP is unset: the pod has no public IP, so it cannot say which host it is}"
+
+export DATA_DIRECTORY="${DATA_DIRECTORY:-/workspace}"
+# The installer is shared between providers and knows none of their names.
+export LLMAO_HOST="$RUNPOD_PUBLIC_IP"
+
+INSTALLER_URL="${LLMAO_INSTALLER_URL:-https://raw.githubusercontent.com/apache/tooling-llmao/refs/heads/main/hosting/install_set.py}"
+STATE="$DATA_DIRECTORY/.llmao"
+CONF_D="$STATE/conf.d"
+SUPERVISORD_CONF="${LLMAO_SUPERVISORD_CONF:-/etc/supervisor/supervisord.conf}"
+RUN_DIR="${LLMAO_RUN_DIR:-/run/llmao}"
+RETRY_FIRST_S="${LLMAO_RETRY_FIRST_S:-15}"
+RETRY_MAX_S="${LLMAO_RETRY_MAX_S:-300}"
+
+provision_once() {
+  # A half-finished earlier attempt leaves units that may not match the set;
+  # start from nothing so a server dropped from the set does not linger.
+  rm -rf "$CONF_D"
+  mkdir -p "$CONF_D"
+  curl -fsSL --max-time 60 -o "$STATE/install_set.py.new" "$INSTALLER_URL" || return 1
+  mv "$STATE/install_set.py.new" "$STATE/install_set.py"
+  SUPERVISOR_CONF_DIR="$CONF_D" VLLM_BIN="$(command -v vllm || echo vllm)" \
+    python3 "$STATE/install_set.py" --no-reload || return 1
+}
+
+mkdir -p "$STATE"
+if [ -f "$STATE/provisioned" ]; then
+  log "already provisioned; fetching nothing"
+else
+  delay="$RETRY_FIRST_S"
+  until provision_once; do
+    log "provisioning failed; retrying in ${delay}s"
+    sleep "$delay"
+    delay=$(( delay * 2 ))
+    if [ "$delay" -gt "$RETRY_MAX_S" ]; then
+      delay="$RETRY_MAX_S"
+    fi
+  done
+  date -u +%Y-%m-%dT%H:%M:%SZ > "$STATE/provisioned"
+  log "provisioned"
+fi
+
+mkdir -p "$RUN_DIR" "$(dirname "$SUPERVISORD_CONF")"
+cat > "$SUPERVISORD_CONF" <<EOF
+; Written by llmao provision.sh on every boot; edits do not survive a restart.
+[unix_http_server]
+file=$RUN_DIR/supervisor.sock
+chmod=0700
+
+[supervisord]
+nodaemon=true
+user=root
+pidfile=$RUN_DIR/supervisord.pid
+logfile=$DATA_DIRECTORY/logs/supervisord.log
+
+[rpcinterface:supervisor]
+supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
+
+[supervisorctl]
+serverurl=unix://$RUN_DIR/supervisor.sock
+
+[include]
+files = $CONF_D/*.conf
+EOF
+mkdir -p "$DATA_DIRECTORY/logs"
+
+exec supervisord -n -c "$SUPERVISORD_CONF"
diff --git a/tests/test_runpod_provision.py b/tests/test_runpod_provision.py
new file mode 100644
index 0000000..4baad3b
--- /dev/null
+++ b/tests/test_runpod_provision.py
@@ -0,0 +1,152 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+"""hosting/runpod/provision.sh under bash, with curl and supervisord stubbed.
+
+The installer it fetches is a stand-in that writes one unit, so these test
+the boot script's own job: provision once, retry until it works, and fetch
+nothing once provisioned.
+"""
+
+import os
+import shutil
+import subprocess
+from pathlib import Path
+
+import pytest
+
+SCRIPT = Path(__file__).resolve().parents[1] / "hosting" / "runpod" / "provision.sh"
+
+# Stands in for install_set.py: proves what it was handed, writes one unit.
+FAKE_INSTALLER = """\
+import os, sys
+assert sys.argv[1:] == ["--no-reload"], sys.argv
+assert os.environ["LLMAO_HOST"] == "198.51.100.8"
+conf = os.environ["SUPERVISOR_CONF_DIR"]
+open(os.path.join(conf, "vllm-model-a.conf"), "w").write("[program:vllm-model-a]\\n")
+"""
+
+# Fails the first $STUB_CURL_FAILS calls, then "downloads" the installer.
+CURL = """\
+#!/bin/bash
+n=$(( $(cat "$STUB_DIR/curl.calls" 2>/dev/null || echo 0) + 1 ))
+echo "$n" > "$STUB_DIR/curl.calls"
+[ "$n" -le "${STUB_CURL_FAILS:-0}" ] && exit 22
+while [ $# -gt 0 ]; do [ "$1" = -o ] && out="$2"; shift; done
+cp "$STUB_DIR/installer.py" "$out"
+"""
+
+SUPERVISORD = """\
+#!/bin/bash
+echo "$*" > "$STUB_DIR/supervisord.args"
+"""
+
+
+@pytest.fixture
+def box(tmp_path):
+    if shutil.which("bash") is None:
+        pytest.skip("no bash")
+    stubs = tmp_path / "stubs"
+    stubs.mkdir()
+    for name, text in (("curl", CURL), ("supervisord", SUPERVISORD)):
+        (stubs / name).write_text(text)
+        (stubs / name).chmod(0o755)
+    (stubs / "installer.py").write_text(FAKE_INSTALLER)
+    workspace = tmp_path / "workspace"
+    env = {
+        "PATH": f"{stubs}:{os.environ['PATH']}",
+        "STUB_DIR": str(stubs),
+        "FLEET_KEY": "fleet-secret",
+        "ASFQUART_URL": "https://llm.example.org",
+        "RUNPOD_PUBLIC_IP": "198.51.100.8",
+        "DATA_DIRECTORY": str(workspace),
+        "LLMAO_SUPERVISORD_CONF": str(tmp_path / "etc" / "supervisord.conf"),
+        "LLMAO_RUN_DIR": str(tmp_path / "run"),
+        "LLMAO_RETRY_FIRST_S": "0",
+    }
+
+    def boot(**extra):
+        return subprocess.run(["bash", str(SCRIPT)], env={**env, **extra}, capture_output=True, text=True, timeout=30)
+
+    return boot, stubs, workspace, tmp_path
+
+
+def _calls(stubs):
+    return int((stubs / "curl.calls").read_text()) if (stubs / "curl.calls").exists() else 0
+
+
+def test_first_boot_provisions_and_execs_supervisord(box):
+    boot, stubs, workspace, tmp_path = box
+    result = boot()
+    assert result.returncode == 0, result.stderr
+    state = workspace / ".llmao"
+    assert (state / "provisioned").exists()
+    assert (state / "conf.d" / "vllm-model-a.conf").exists()
+    conf = tmp_path / "etc" / "supervisord.conf"
+    assert (stubs / "supervisord.args").read_text().split() == ["-n", "-c", str(conf)]
+    text = conf.read_text()
+    assert f"files = {state / 'conf.d'}/*.conf" in text
+    # The socket is on the container disk, never the volume.
+    assert f"file={tmp_path / 'run'}/supervisor.sock" in text
+
+
+def test_provisioned_boot_fetches_nothing(box):
+    boot, stubs, workspace, _ = box
+    assert boot().returncode == 0
+    assert _calls(stubs) == 1
+    result = boot()
+    assert result.returncode == 0, result.stderr
+    assert _calls(stubs) == 1
+    assert "already provisioned" in result.stderr
+    # The units survive: a restart serves the same set.
+    assert (workspace / ".llmao" / "conf.d" / "vllm-model-a.conf").exists()
+
+
+def test_retries_until_the_fetch_succeeds(box):
+    boot, stubs, workspace, _ = box
+    result = boot(STUB_CURL_FAILS="2")
+    assert result.returncode == 0, result.stderr
+    assert _calls(stubs) == 3
+    assert result.stderr.count("provisioning failed") == 2
+    assert (workspace / ".llmao" / "provisioned").exists()
+
+
+def test_a_failing_installer_is_retried_and_leaves_no_marker_until_it_works(box):
+    boot, stubs, workspace, _ = box
+    good = (stubs / "installer.py").read_text()
+    # Fails on its first run, as /vllm/config does before the fleet.hosts row exists.
+    (stubs / "installer.py").write_text(
+        "import os, sys\n"
+        "flag = os.path.join(os.environ['STUB_DIR'], 'ran')\n"
+        "if not os.path.exists(flag):\n"
+        "    open(flag, 'w').close()\n"
+        "    sys.exit('config fetch HTTP 403')\n" + good
+    )
+    result = boot()
+    assert result.returncode == 0, result.stderr
+    assert result.stderr.count("provisioning failed") == 1
+    assert (workspace / ".llmao" / "provisioned").exists()
+
+
+def test_no_public_ip_refuses_to_start(box):
+    boot, stubs, workspace, _ = box
+    result = boot(RUNPOD_PUBLIC_IP="")
+    assert result.returncode != 0
+    assert "RUNPOD_PUBLIC_IP" in result.stderr
+    assert not (workspace / ".llmao" / "provisioned").exists()
+    assert _calls(stubs) == 0


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org