You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/10/06 01:53:53 UTC
(tooling-llmao) 06/23: allowance from the spend log, contention sampling, and a usage page
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-llmao.git
commit d7ea8d101e64398095101dd55d29ea3e31c16fb5
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Mon Oct 5 20:22:41 2026 +0000
allowance from the spend log, contention sampling, and a usage page
Three pieces, aimed at next week's pilot.
ALLOWANCE, DERIVED NOT STORED
backend.allowance(user) builds the rolling buckets from LiteLLM's own spend
rows on read, cached for 60s. Two reasons that beats a stored counter.
It cannot drift from actual usage, because it IS actual usage. A stored
counter can disagree with the log after a crash, a double-count or a replay,
and the disagreement stays invisible until someone audits it.
And there is nothing to migrate when this is replaced. The design this stands
in for -- defer a free-tier call only when projects need the box -- needs no
allowance at all, so a stored one would be a table to drop and a UI to unwind.
Derived, it just stops being called.
The cost is a query per principal per TTL, which is cheap on a pilot-sized
log. At scale it wants LiteLLM's daily aggregate table or a materialised
counter; the shape here does not change if that happens.
CONTENTION SAMPLING
llmao/contention.py polls each box's /metrics and logs one JSON line per box
per interval: running, waiting, kv_pct, preempted.
The pilot exists to answer a question the spend log cannot: do free-tier and
project work ever contend for a GPU? The log records what ran, not what
waited, and queue depth is only visible while it is happening.
If waiting stays at zero throughout, the weekly allowance is solving a problem
that is not there and the replacement needs no allowance at all. If it does
not, we have the numbers to size the real thing.
Log lines rather than a table: a pilot is weeks, the volume is a few thousand
lines, and grep and jq answer every question we have now. A schema designed
before the questions are known is a schema to redo.
THE USAGE PAGE
/usage shows a committer their free-tier draw, how much is left, and WHEN
MORE RETURNS -- 'tomorrow', 'on Tuesday'. That last part is the reason for
daily buckets over a live aggregate: 'you are out' is a dead end, and 'you are
out, 200k returns tomorrow' is a plan.
Warns at 80% rather than only at the wall. Discovering a limit by hitting it
mid-job is the experience worth designing away.
Free-tier capacity and project dollars render as separate figures and are
never summed. Tokens on self-hosted models are a share of hardware already
paid for; dollars on commercial models are money that did not exist a second
ago. One column would invite adding them.
---
llmao/contention.py | 183 ++++++++++++++++++++++++++++++++++++++++++++++++
llmao/litellm_client.py | 118 +++++++++++++++++++++++++++++++
llmao/seam.py | 44 ++++++++++++
pages.py | 66 +++++++++++++++++
templates/usage.ezt | 80 +++++++++++++++++++++
5 files changed, 491 insertions(+)
diff --git a/llmao/contention.py b/llmao/contention.py
new file mode 100644
index 0000000..f25f54e
--- /dev/null
+++ b/llmao/contention.py
@@ -0,0 +1,183 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Sample fleet contention, so the pilot answers a question.
+
+WHY THIS EXISTS
+
+The weekly token allowance is a proxy. What we actually want to know is
+whether free-tier work and project work ever contend for a GPU -- because if
+they rarely do, the allowance is solving a problem that is not there, and the
+replacement design (defer a free-tier call only when projects need the box)
+needs no allowance at all.
+
+Nobody can answer that from the spend log: it records what ran, not what
+waited. Queue depth is only visible while it is happening, so it has to be
+sampled.
+
+WHAT IT RECORDS
+
+One line per box per interval, as structured JSON on the logger. Not a
+database: a pilot is weeks, the volume is a few thousand lines, and grep and
+jq answer every question we currently have. A table would be a schema to
+design before knowing what the questions are.
+
+ {"ts": ..., "box": "gemma4-26b", "running": 4, "waiting": 2,
+ "kv_pct": 31.0, "preempted": 0}
+
+The three that matter:
+
+ waiting > 0 while running > 0 contention is real and measurable
+ waiting == 0 throughout the allowance is unnecessary; capacity is
+ not the scarce thing
+ preempted climbing vLLM is evicting under memory pressure,
+ which is a different problem from queueing
+ and wants a different fix
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import logging
+import time
+from dataclasses import dataclass
+from typing import Any
+
+import httpx
+
+_LOGGER = logging.getLogger("llmao.contention")
+
+# Metric names, with the vllm: prefix stripped at parse time.
+_WANTED = {
+ "num_requests_running": "running",
+ "num_requests_waiting": "waiting",
+ "gpu_cache_usage_perc": "kv_pct",
+ "num_preemptions": "preempted",
+ "num_preemptions_total": "preempted",
+}
+
+
+@dataclass
+class Sample:
+ ts: float
+ box: str
+ running: float = 0.0
+ waiting: float = 0.0
+ kv_pct: float = 0.0
+ preempted: float = 0.0
+
+ def as_dict(self) -> dict[str, Any]:
+ return {
+ "ts": round(self.ts, 1),
+ "box": self.box,
+ "running": self.running,
+ "waiting": self.waiting,
+ "kv_pct": round(self.kv_pct * 100, 1) if self.kv_pct <= 1 else round(self.kv_pct, 1),
+ "preempted": self.preempted,
+ }
+
+
+def parse_metrics(box: str, text: str, *, now: float | None = None) -> Sample:
+ """Pull the four figures out of a Prometheus exposition response.
+
+ Hand-parsed rather than via a client library: four metrics, one format,
+ and a dependency for this would be heavier than the code.
+ """
+ out = Sample(ts=now if now is not None else time.time(), box=box)
+ for line in text.splitlines():
+ if not line or line.startswith("#"):
+ continue
+ name, _, rest = line.partition("{")
+ if not rest:
+ name, _, value = line.partition(" ")
+ else:
+ _, _, value = rest.partition("} ")
+ field = _WANTED.get(name.removeprefix("vllm:"))
+ if not field:
+ continue
+ try:
+ setattr(out, field, float(value.strip()))
+ except (TypeError, ValueError):
+ continue
+ return out
+
+
+class ContentionSampler:
+ """Polls each box's /metrics and logs a line per sample.
+
+ Failures are logged once per box per outage rather than every interval:
+ a box being down is worth knowing and is not worth a line every fifteen
+ seconds for an hour.
+ """
+
+ def __init__(self, fleet: Any, *, interval_s: int = 15, timeout_s: float = 3.0):
+ self._fleet = fleet
+ self._interval = interval_s
+ self._timeout = timeout_s
+ self._down: set[str] = set()
+ self._task: asyncio.Task | None = None
+
+ def _targets(self) -> list[tuple[str, str]]:
+ """(box name, metrics url) for every self-hosted deployment serving."""
+ out = []
+ for dep in getattr(self._fleet, "deployments", None) or []:
+ if not getattr(dep, "self_hosted", False):
+ continue
+ base = getattr(dep, "api_base", None)
+ if not base:
+ continue
+ root = base.rstrip("/")
+ if root.endswith("/v1"):
+ root = root[: -len("/v1")]
+ out.append((getattr(dep, "model_name", getattr(dep, "name", "?")),
+ f"{root}/metrics"))
+ return out
+
+ async def sample_once(self, client: httpx.AsyncClient) -> list[Sample]:
+ samples = []
+ for box, url in self._targets():
+ try:
+ resp = await client.get(url, timeout=self._timeout)
+ resp.raise_for_status()
+ except Exception as e:
+ if box not in self._down:
+ self._down.add(box)
+ _LOGGER.warning("contention: %s unreachable (%s)", box,
+ type(e).__name__)
+ continue
+ self._down.discard(box)
+ samples.append(parse_metrics(box, resp.text))
+ return samples
+
+ async def run(self) -> None:
+ async with httpx.AsyncClient() as client:
+ while True:
+ try:
+ for s in await self.sample_once(client):
+ # One JSON object per line: greppable, jq-able, and
+ # loadable into pandas without a parser.
+ _LOGGER.info("contention %s", json.dumps(s.as_dict()))
+ except asyncio.CancelledError:
+ raise
+ except Exception:
+ _LOGGER.exception("contention: sample failed")
+ await asyncio.sleep(self._interval)
+
+ def start(self) -> asyncio.Task:
+ self._task = asyncio.create_task(self.run(), name="contention-sampler")
+ return self._task
diff --git a/llmao/litellm_client.py b/llmao/litellm_client.py
index c707699..4cba7b2 100644
--- a/llmao/litellm_client.py
+++ b/llmao/litellm_client.py
@@ -33,8 +33,12 @@ from dataclasses import dataclass
from typing import Any, Protocol
from urllib.parse import urlparse
+import datetime as _dt
+import time
+
import httpx
+from llmao.token_window import RollingAllowance
from llmao.fleet import ExtraLiteLLM, snapshot_for_status
from llmao.model_status import (
HEALTHY,
@@ -231,6 +235,8 @@ class Backend(Protocol):
) -> CreatedKey: ...
async def delete_key(self, token_id: str) -> None: ...
async def usage(self, project: str | None) -> list[dict]: ...
+ async def allowance(self, user: str, *, tier: str = TIER_FREE) -> RollingAllowance: ...
+ def invalidate_allowance(self, user: str | None = None) -> None: ...
async def aclose(self) -> None: ...
@@ -293,6 +299,10 @@ class LiteLLMBackend:
self._cfg = cfg
self.fleet = fleet
self._team_ids: dict[str, str] = {} # project → team_id
+ # user → (monotonic stamp, daily token buckets). Allowances are
+ # derived from the spend log on read; this keeps a page refresh from
+ # costing a scan.
+ self._allowance_cache: dict[str, tuple[float, dict[str, int]]] = {}
base = cfg.litellm.base_url.rstrip("/") + "/"
timeout_s = int(cfg.litellm.request_timeout_s)
self._client = httpx.AsyncClient(
@@ -509,6 +519,10 @@ class LiteLLMBackend:
)
self._raise_http(resp)
_LOGGER.info("tier %s updated: %s", tier, sorted(changes))
+ # The cap is re-read on every check, but cached buckets are not --
+ # without this an admin lowering a cap sees a stale remaining figure
+ # for up to a minute and concludes the control did not work.
+ self.invalidate_allowance()
return await self.tier_entitlement(tier)
async def update_team_entitlement(
@@ -732,6 +746,110 @@ class LiteLLMBackend:
# Spend APIs not wired yet.
return []
+ # Allowance reads are cached: a committer refreshing a page should not
+ # cost a spend-log scan each time. Short enough that a long job's draw
+ # shows up while it is still running.
+ _ALLOWANCE_TTL_S = 60
+
+ async def allowance(self, user: str, *, tier: str = TIER_FREE) -> RollingAllowance:
+ """A principal's rolling allowance, derived from the spend log.
+
+ NOT stored. The buckets are computed from LiteLLM's own spend rows on
+ read, which has two properties worth the query cost:
+
+ The allowance cannot drift from actual usage, because it IS actual
+ usage. A stored counter can disagree with the log after a crash, a
+ double-count or a replay, and the disagreement is invisible until
+ somebody audits it.
+
+ And there is nothing to migrate when this is replaced. The design
+ this is standing in for -- defer a free-tier call when projects need
+ the box -- needs no allowance at all, so a stored one would be a
+ table to drop and a UI to unwind. Derived, it just stops being
+ called.
+
+ The cost is a query per principal per TTL. On a pilot-sized spend log
+ that is cheap; at scale it wants either LiteLLM's daily aggregate
+ table or a materialised counter, and the shape here does not change
+ if that happens.
+ """
+ user = (user or "").strip()
+ if not user:
+ raise BackendUnavailableError("allowance requires a user")
+
+ limits = await self.tier_entitlement(tier)
+ cap = int(limits.get("token_cap") or 0)
+ window = int(limits.get("token_window_days") or 7)
+ if cap <= 0:
+ # No cap configured for this tier: an allowance with no ceiling
+ # is not meaningful, so say so rather than returning one that
+ # silently never exhausts.
+ raise BackendUnavailableError(
+ f"tier {tier} has no token_cap; nothing to meter against"
+ )
+
+ cached = self._allowance_cache.get(user)
+ if cached is not None:
+ at, rows = cached
+ if (time.monotonic() - at) < self._ALLOWANCE_TTL_S:
+ return RollingAllowance(cap=cap, window_days=window,
+ buckets=dict(rows))
+
+ buckets = await self._spend_buckets(user, window)
+ self._allowance_cache[user] = (time.monotonic(), dict(buckets))
+ return RollingAllowance(cap=cap, window_days=window, buckets=buckets)
+
+ async def _spend_buckets(self, user: str, window_days: int) -> dict[str, int]:
+ """Daily token totals for one user over the window.
+
+ Counts TOTAL tokens -- prompt plus completion -- because the
+ allowance is a share of the fleet and both halves consume it. Prefill
+ is cheaper per token than decode, which is why the DOLLAR figures are
+ weighted and this one is not: an allowance is a quota, not a bill.
+ """
+ start = (_dt.datetime.now(_dt.UTC).date()
+ - _dt.timedelta(days=window_days - 1))
+ resp = await self._request(
+ "GET", "spend/logs",
+ params={"user_id": user, "start_date": start.isoformat()},
+ )
+ self._raise_http(resp)
+ rows = resp.json()
+ if isinstance(rows, dict):
+ rows = rows.get("data") or rows.get("logs") or []
+ if not isinstance(rows, list):
+ return {}
+
+ buckets: dict[str, int] = {}
+ for row in rows:
+ if not isinstance(row, dict):
+ continue
+ stamp = row.get("startTime") or row.get("start_time")
+ if not stamp:
+ continue
+ day = str(stamp)[:10]
+ total = row.get("total_tokens")
+ if total is None:
+ total = (row.get("prompt_tokens") or 0) + (row.get("completion_tokens") or 0)
+ try:
+ buckets[day] = buckets.get(day, 0) + int(total)
+ except (TypeError, ValueError):
+ continue
+ return buckets
+
+ def invalidate_allowance(self, user: str | None = None) -> None:
+ """Drop cached allowance rows.
+
+ Called after a tier change, since the cap is re-read on every check
+ but the cached buckets are not. Without this an admin lowering a cap
+ sees the old remaining figure for up to a minute, which looks like
+ the control not working.
+ """
+ if user is None:
+ self._allowance_cache.clear()
+ else:
+ self._allowance_cache.pop(user, None)
+
def deployment_body(self, dep) -> dict:
"""POST /model/new payload from models.yaml + this deployment's api_base."""
entry = self.fleet.models.get(dep.model_name)
diff --git a/llmao/seam.py b/llmao/seam.py
index e44459e..48ab5c5 100644
--- a/llmao/seam.py
+++ b/llmao/seam.py
@@ -234,6 +234,50 @@ class Seam:
async def list_my_keys(self, identity: Identity) -> list[KeyInfo]:
return await self._backend.list_keys(user=identity.uid, size=100)
+ async def my_allowance(self, identity: Identity):
+ """This committer's rolling free-tier allowance.
+
+ No authz decision: a principal always sees their own draw. Project
+ figures go through the normal membership checks below.
+ """
+ return await self._backend.allowance(identity.uid)
+
+ async def my_project_usage(self, identity: Identity) -> list[dict]:
+ """Per-project draw for the projects this committer belongs to.
+
+ Tokens and dollars are kept as separate fields deliberately. Tokens on
+ self-hosted models are a share of capacity the Foundation has already
+ paid for; dollars on commercial models are money that did not exist a
+ second ago. Summing them would produce a figure with no meaning, and a
+ single column invites exactly that.
+ """
+ out = []
+ for project in sorted(identity.committees):
+ try:
+ rows = await self._backend.usage(project)
+ except BackendUnavailableError:
+ continue
+ tokens = 0
+ spend = 0.0
+ for row in rows or []:
+ if not isinstance(row, dict):
+ continue
+ if row.get("user") not in (None, identity.uid):
+ continue
+ tokens += int(row.get("total_tokens") or 0)
+ # Only commercial rows carry real money. Self-hosted spend is
+ # priced from measured throughput for reporting, and adding it
+ # here would put capacity into a column labelled dollars.
+ if row.get("cost_basis") != "derived_capacity":
+ spend += float(row.get("total_cost_usd") or 0.0)
+ out.append({
+ "name": project,
+ "tokens_h": f"{tokens / 1_000_000:.1f}M" if tokens >= 1_000_000
+ else (f"{tokens // 1_000}k" if tokens >= 1000 else str(tokens)),
+ "spend_h": f"${spend:,.2f}",
+ })
+ return out
+
@require_admin
async def list_automation_keys(self, identity: Identity, project: str) -> list[KeyInfo]:
"""Automation keys for a project (admin / PMC only)."""
diff --git a/pages.py b/pages.py
index 7177c73..b76abb0 100644
--- a/pages.py
+++ b/pages.py
@@ -720,6 +720,72 @@ async def keys_list(result):
return result
+def _tokens_h(n: int) -> str:
+ """Human token count. 340k reads faster than 340,000 and this is a glance."""
+ n = int(n or 0)
+ if n >= 1_000_000:
+ return f"{n / 1_000_000:.1f}M".replace(".0M", "M")
+ if n >= 1_000:
+ return f"{n // 1_000}k"
+ return str(n)
+
+
+def _when_h(day) -> str:
+ """'tomorrow' / 'Tuesday' / a date, whichever is most useful.
+
+ 'Your tokens return 2026-09-23' is technically complete and makes the
+ reader do arithmetic. The point of showing a return date at all is to turn
+ a dead end into a plan.
+ """
+ import datetime as dt
+ today = dt.datetime.now(dt.UTC).date()
+ delta = (day - today).days
+ if delta <= 0:
+ return "today"
+ if delta == 1:
+ return "tomorrow"
+ if delta < 7:
+ return f"on {day.strftime('%A')}"
+ return f"on {day.isoformat()}"
+
+
+@APP.get("/usage")
+@asfquart.auth.require
+@APP.use_template(TEMPLATES / "usage.ezt")
+@page(title="My usage")
+async def usage_mine(result):
+ """What this committer has drawn, and when more becomes available.
+
+ The free-tier figure is a capacity share and the project figures include
+ real money, so the template keeps them visually apart. Adding them would
+ produce a number that means nothing.
+ """
+ ident = await current_identity(APP.cfg)
+ allowance = await APP.seam.my_allowance(ident)
+
+ result.cap_h = _tokens_h(allowance.cap)
+ result.used_h = _tokens_h(allowance.used())
+ result.remaining_h = _tokens_h(allowance.remaining())
+ result.window_days = allowance.window_days
+ result.pct = min(100, int(allowance.used() / allowance.cap * 100)) if allowance.cap else 0
+ result.exhausted = ezt.boolean(allowance.exhausted())
+ # Warn before the wall rather than at it: discovering the limit by
+ # hitting it mid-job is the experience worth avoiding.
+ result.nearly = ezt.boolean(result.pct >= 80)
+
+ returns = allowance.returns()
+ if returns:
+ day, amount = returns[0]
+ result.next_return = _when_h(day)
+ result.next_return_amount_h = _tokens_h(amount)
+ else:
+ result.next_return = None
+ result.next_return_amount_h = None
+
+ result.projects = await APP.seam.my_project_usage(ident)
+ return result
+
+
@APP.get("/keys/other")
@asfquart.auth.require
@APP.use_template(TEMPLATES / "keys_other.ezt")
diff --git a/templates/usage.ezt b/templates/usage.ezt
new file mode 100644
index 0000000..793d76b
--- /dev/null
+++ b/templates/usage.ezt
@@ -0,0 +1,80 @@
+[include "header.ezt"]
+<div class="container py-4">
+ <div class="mt-2">[include "flashes.ezt"]</div>
+
+ <h1 class="h3 mb-3">My usage</h1>
+
+ <p class="text-muted small">
+ Free-tier work runs on ASF-hosted models and shares the fleet with every
+ other committer. Heavy or sustained work belongs on a project allocation.
+ </p>
+
+ <div class="card mb-4">
+ <div class="card-body">
+ <div class="d-flex justify-content-between align-items-baseline mb-1">
+ <span class="fw-semibold">Free tier</span>
+ <span class="text-muted small">
+ [used_h] of [cap_h] tokens · rolling [window_days] days
+ </span>
+ </div>
+ <div class="progress" style="height:10px" role="progressbar"
+ aria-valuenow="[pct]" aria-valuemin="0" aria-valuemax="100"
+ aria-label="Free tier usage">
+ <div class="progress-bar[if-any exhausted] bg-danger[else][if-any nearly] bg-warning[end][end]"
+ style="width:[pct]%"></div>
+ </div>
+
+ [if-any exhausted]
+ <p class="small mt-2 mb-0">
+ <span class="fw-semibold">You are out for now.</span>
+ [if-any next_return]
+ [next_return_amount_h] tokens return [next_return].
+ [end]
+ </p>
+ [else]
+ <p class="text-muted small mt-2 mb-0">
+ [remaining_h] left.
+ [if-any next_return]
+ [next_return_amount_h] more returns [next_return].
+ [end]
+ </p>
+ [end]
+ </div>
+ </div>
+
+ [if-any projects]
+ <h2 class="h5 mb-2">Projects</h2>
+ <div class="table-responsive">
+ <table class="table table-sm align-middle">
+ <thead>
+ <tr>
+ <th>Project</th>
+ <th class="text-end">Tokens</th>
+ <th class="text-end">Commercial spend</th>
+ </tr>
+ </thead>
+ <tbody>
+ [for projects]
+ <tr>
+ <td><a href="/projects/[projects.name]">[projects.name]</a></td>
+ <td class="text-end">[projects.tokens_h]</td>
+ <td class="text-end">[projects.spend_h]</td>
+ </tr>
+ [end]
+ </tbody>
+ </table>
+ </div>
+ <p class="text-muted small">
+ Commercial spend is real money against the project’s budget. Tokens
+ on ASF-hosted models are a share of fleet capacity, not a bill — the
+ hardware costs the same whether or not it is busy.
+ </p>
+ [else]
+ <p class="text-muted small">
+ You are not on a project allocation. Project keys draw on a PMC’s
+ share and can reach commercial models; ask your PMC if you need one.
+ </p>
+ [end]
+
+</div>
+[include "footer.ezt"]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org