[I] Work plan - Sep 02 (tooling-llmao)

["andrewmusselman (via GitHub)" <gi...@apache.org>]

andrewmusselman opened a new issue, #2:
URL: https://github.com/apache/tooling-llmao/issues/2

   # llmao — work plan
   
   One ordered list. Each entry is a sub-issue of this issue; a `review` sub-issue explains
   the reasoning behind them and does not sequence anything.
   
   Phases are ordered by dependency, not by size. **p6 is a parallel track** and
   can start at any point.
   
   ---
   
   ## Phase 0 — make it work at all
   
   Nothing routes through the gateway today. All four are small.
   
   | | why now |
   |---|---|
   | **Stopgap: add api_base to model_list.yaml** | two lines; models answer today rather than 401ing to OpenAI |
   | **AuthzError re-renders the page template** | authz failures return a full traceback including the session dict. Also unblocks `/fleet`, which is where skew is visible |
   | **max_model_len absent from the example catalog** | one line per entry; removes a shipped hang condition |
   | **LiteLLM proxy publicly reachable over plaintext** | vhost change: proxy `/v1/*` to loopback, close 4000 |
   
   The first is deliberately a stopgap. It is replaced in phase 2, not extended.
   
   ---
   
   ## Phase 1 — trust the health signal before building on it
   
   Phase 2 hangs entirely off health transitions: a route is created on the
   healthy edge and deleted on the down edge. If those fire wrongly, it either
   withholds a working server or pulls a live one from rotation.
   
   | | why now |
   |---|---|
   | **Fleet health code is untested against real hardware** | described as "a rough v1 from Grok… needs to be tested in reality". Exercise a box that never comes up, one that dies, and one that flaps |
   | **Surface config skew on the /fleet page** | `srv.skew` is already populated and logged; nothing displays it. This is why the routing mismatch went unnoticed. Needs the authz fix from phase 0 |
   | **`db_model: false` on a route created via /model/new** | cheap to pin down, and phase 2 needs to distinguish llmao-created routes from seeded ones |
   
   Skipping this phase is possible. It means finding out in phase 2 whether the
   transitions were right, with more moving parts in the way.
   
   ---
   
   ## Phase 2 — fleet state into LiteLLM
   
   **One commit, three parts.** Landing any part alone puts routes with no
   `api_base` back into the proxy — which is the phase 0 outage, reproduced.
   
   | | |
   |---|---|
   | **Move fleet state into LiteLLM, enable STORE_MODEL_IN_DB** | routes carry host, port, key and recipe |
   | **Health-gated route registration** | a route exists only while its vLLM is serving |
   | **Remove the model_list include from litellm.yaml** | otherwise LiteLLM reads catalog entries as routes |
   
   Verified already: `model_info` survives a `/model/new` round-trip intact.
   `litellm_params` reads back **encrypted**, so `api_base` cannot identify a
   route's host — duplicate the host into `model_info` or decrypt via LiteLLM's
   accessor. Decide that before writing code.
   
   Also settle where pending assignments live between "host added" and "vLLM
   healthy". Preferred: a small table in the same Postgres.
   
   `model_list.yaml.example` in this packet is the post-change version. **Do not
   copy it before this phase** — it has no `api_base`, which is correct afterwards
   and broken before.
   
   ---
   
   ## Phase 3 — sizing and observability
   
   Independent of each other; any order.
   
   | | why |
   |---|---|
   | **Fit validation from model requirements** | `validate_fleet` does not check a placement fits. A percentage cannot be right on two card sizes |
   | **Report observed vLLM state back to the control plane** | the box knows its measured KV cache and actual GPU; llmao never hears it. Every sizing problem so far was diagnosed by reading a startup log by hand |
   | **Use --kv-cache-memory instead of a utilisation percentage** | vLLM prints the exact byte count. Measured beats derived |
   
   ---
   
   ## Phase 4 — UI
   
   | | |
   |---|---|
   | **Fleet management in the UI** | add, retire, edit. Needs phase 2 |
   
   ---
   
   ## Parallel track — p6
   
   Can start now. Nothing above blocks it, and it blocks nothing above.
   
   | | |
   |---|---|
   | **Puppet manifest for the llmao gateway host** | portal as a `pipservice`, LiteLLM as a small manifest, Postgres from `base_packages` |
   | **Backup for the LiteLLM database** | `$dest` is unresolved — confirm what BackupPC actually captures |
   
   Two open questions in `p6/NOTES.md`: whether `update_pipenv` handles `uv` (the
   script suggests yes), and the `/opt/llmao/llmao.yaml` vs `config.yaml`
   filename, being resolved by renaming.
   
   ---
   
   ## Whenever
   
   Small, unblocked, no dependencies.
   
   | | |
   |---|---|
   | **`make db` rotates the Postgres password on every run** | running it twice silently breaks a working `litellm.yaml` |
   | **Rotate the self-hosted vLLM API key** | pasted around during debugging; never committed |
   | **SSL_VERIFY=0 is a stopgap that will outlive its reason** | file it or make it warn |
   | **Two instances of one model do not need artificial model names** | documentation; prevents a worse catalog shape |
   
   ---
   
   ## The one thing to get right
   
   Phase 2 is a single change. The temptation is to land the route push first and
   tidy up afterwards — that reproduces the outage on every restart, and it will
   look like a regression rather than an incomplete migration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org