[I] Model names and caps: where things stand (tooling-llmao)

["gstein (via GitHub)" <gi...@apache.org>]

gstein opened a new issue, #52:
URL: https://github.com/apache/tooling-llmao/issues/52

   Status as of 2026-10-13, at commit `9a6c82f`. Two topics that touch each
   other: what a model is called when more than one provider serves it, and
   what limits a key or a project. Each section says what is built and what is
   still to decide.
   
   ## 1. One model, several providers
   
   ### What is built
   
   - Every model in the fleet today runs on our own boxes. Nothing is bought
     in yet, so no real money is involved.
   - LiteLLM records usage per model **name**. It does not record which
     deployment served a request in a form we can total by.
   - Cost on our own boxes is an estimate: a share of capacity already paid
     for, derived from throughput. Cost from a provider is a bill.
   - Each deployment knows which of the two it is
     (`FleetDeployment.cost_basis`). A model name is asked through
     `Fleet.cost_basis(model_name)`: it is an estimate only when every
     deployment of that name is on our boxes. A name served both ways, or by
     nothing, is treated as real money, so a bill is never shown as an
     estimate.
   - Deployments on our boxes are tagged with a LiteLLM access group,
     `llmao:self-hosted`. Keys with no project sit in one of two pretend
     projects (`llmao:project:free-tier`, `llmao:project:service`), and those
     are limited to that group. That is what stops a free key from spending
     money once a paid provider is added.
   
   ### The problem
   
   If one name is served from our boxes and also from a provider, its usage
   and cost cannot be split into estimate and bill. The whole name has to be
   called real money, which overstates the bill.
   
   ### The option on the table
   
   Give each provider its own name for the model, for example `qwen-hosted`
   and `qwen-external`, both carrying the same Hugging Face id. Usage, limits
   and cost then separate by name, and the Hugging Face id says they are the
   same model.
   
   ### To decide
   
   - Whether to synthesise names this way.
   - How a person picks between them, or whether routing picks for them.
   - What the Models page shows: one row per model, or one per name.
   - `models.yaml` marks a model `self_hosted`. Where a model runs is a fact
     about a deployment, not the model, so that flag has to leave
     `models.yaml` whichever way this goes.
   
   ### Known gaps
   
   - Deployments registered in LiteLLM before the access group existed do not
     carry the tag yet. Until they do, the group is empty.
   - The restriction has not been exercised against the deployed service.
   
   ## 2. Caps
   
   There are three kinds of limit, and they are held in three places.
   
   | Limit | Applies to | Held in | Enforced by |
   |---|---|---|---|
   | Requests/minute, tokens/minute, parallel requests | a key, by its tier | the key in LiteLLM, set when it is minted | LiteLLM |
   | Dollar budget | a project | the LiteLLM team | LiteLLM, and only where dollars are real |
   | Tokens per rolling window | a project, or a person on the free tier | see below | nothing: counted and shown |
   
   ### Tiers
   
   A key has one of three tiers: `free_tier` (personal, no project),
   `project`, or `service` (pipelines). Starting limits for each are in
   `config.yaml` under `entitlements`. A site admin can override them from the
   Admin page; overrides are one row in LiteLLM's `/v1/memory`
   (`llmao:tiers`). A new key gets its tier's limits, config plus overrides,
   at the moment it is minted.
   
   ### Token caps
   
   - **Free tier:** 1,000,000 tokens over a rolling 7 days per person, from
     config or the admin override. Only usage through keys in the free-tier
     pretend project counts towards it.
   - **Projects:** each project has `token_cap` and `token_window_days`, held
     as metadata on its LiteLLM team, starting at 20,000,000 a week. A PMC
     member asks for more through a capacity request, which a site admin
     approves.
   - The window rolls. It is a sum over daily buckets, with no reset day.
   - Usage comes from LiteLLM's `/user/daily/activity`.
   - A token cap is counted and shown on My Keys and the project page. Nothing
     stops a key when the cap is reached. The per-minute limits are the only
     thing that holds a key back on our own boxes.
   
   ### To decide
   
   - Whether per-project caps stay in LiteLLM team metadata or move to
     `/v1/memory`, where access requests and tier overrides already went. In
     team metadata they can be changed from LiteLLM's own admin interface
     without LLMAO seeing it, and reading them costs one call per project.
   - Whether a token cap should ever stop a key, and what does the stopping.
   - With synthesised names: whether a cap counts `qwen-hosted` and
     `qwen-external` together or apart.
   
   ### Known gaps
   
   - A key keeps the limits it was minted with. If its tier's limits change
     later, the key is out of line until it is revoked and reissued. Showing
     such keys to their owners is planned and not built.
   - A background check now records when a pretend project has been widened
     outside LLMAO, and a site admin can put it back from the Admin page. It
     does not yet look at team budgets or caps.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org