You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by ak...@apache.org on 2026/09/13 02:01:50 UTC
(tooling-llmao) branch main updated: llmao-saturate: size prompts per model, and stop misreading the verdict
This is an automated email from the ASF dual-hosted git repository.
andrewmusselman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-llmao.git
The following commit(s) were added to refs/heads/main by this push:
new 9f60776 llmao-saturate: size prompts per model, and stop misreading the verdict
9f60776 is described below
commit 9f607763956214d9af569ee8986a31f6504276ad
Author: Andrew Musselman <ak...@apache.org>
AuthorDate: Sat Sep 12 19:01:37 2026 -0700
llmao-saturate: size prompts per model, and stop misreading the verdict
Two things made the output wrong rather than merely imprecise.
The prompt size was fixed at 32,000 tokens, which exceeds the 8B's entire
40,960-token window -- so --all failed on it with a 400 every run instead of
measuring anything. It now defaults to a third of each model's window, read
from the box at runtime.
And the verdict told every queueing box to raise --max-num-batched-tokens,
including boxes already running 16384. It now distinguishes the two cases:
only 2-3 requests ever running is the signature of the 2048 default, and
raising it is the fix; many running and still queueing means the budget is
already raised and the limit is the card's prefill throughput -- a faster
card would help, a larger one would not.
Also prints what the box is actually serving before the table -- utilisation,
KV dtype, cache size, and vLLM's own concurrency estimate -- so the reader
can judge the verdict rather than take it on trust.
---
bin/llmao-saturate | 91 +++++++++++++++++++++++++++++++++++++++++++++++++-----
1 file changed, 83 insertions(+), 8 deletions(-)
diff --git a/bin/llmao-saturate b/bin/llmao-saturate
old mode 100755
new mode 100644
index 45182f0..fdb8e4f
--- a/bin/llmao-saturate
+++ b/bin/llmao-saturate
@@ -3,12 +3,20 @@
export VLLM_API_KEY=...
llmao-saturate --model qwen3.8-27b
- llmao-saturate --all --prompt-tokens 32000
+ llmao-saturate --all # sizes prompts per model
+ llmao-saturate --model qwen3-8b --prompt-tokens 16000
llmao-saturate --model qwen3-8b --levels 8,16,32,64
Hits the boxes directly. LiteLLM in the path adds its own limits and
confuses the question.
+PROMPT SIZE IS DERIVED, NOT FIXED
+
+By default each model gets prompts of one third its own window. A fixed size
+is wrong across a fleet: 32,000 tokens exceeds the 8B's entire 40,960-token
+window, so --all failed on it with a 400 every time instead of measuring
+anything.
+
WHY LARGE PROMPTS
KV cache is consumed by CONTEXT, not by request count. Eight 60-token
@@ -34,7 +42,12 @@ HOW TO READ IT
large prompts something is almost always
momentarily queued. Not a finding.
- Sustained queue, KV% low the SCHEDULER is the limit.
+ Sustained queue, KV% low, the 2048 default. Raise
+ only 2-3 running --max-num-batched-tokens.
+
+ Sustained queue, KV% low, the batch budget is already raised; this is
+ many running prefill throughput. A FASTER card would
+ help, a larger one would not.
Raise --max-num-batched-tokens. The default
2048 held every box here to 2-3 concurrent
while 90% of memory sat idle; 16384 took the
@@ -79,6 +92,35 @@ from _discover import direct_endpoints # noqa: E402
# The window comes from the box itself at runtime, not from a table here:
# it changes whenever a box is relaunched with a different --max-model-len,
# and a stale value would silently send prompts that cannot fit.
+def served_config(hostport: str, key: str) -> dict:
+ """What the box is actually running, from its own /metrics.
+
+ The verdict is only useful if it knows the current settings. Telling
+ someone to raise a value that is already raised sends them chasing a
+ setting that is not the problem.
+ """
+ req = urllib.request.Request(
+ f"http://{hostport}/metrics",
+ headers={"Authorization": f"Bearer {key}"})
+ out: dict = {}
+ try:
+ with urllib.request.urlopen(req, timeout=10) as r:
+ text = r.read().decode()
+ except Exception:
+ return out
+ for line in text.splitlines():
+ if not line.startswith("vllm:cache_config_info"):
+ continue
+ start, end = line.find("{"), line.rfind("}")
+ if start < 0:
+ continue
+ for part in line[start + 1:end].split(","):
+ k, _, v = part.partition("=")
+ out[k.strip()] = v.strip().strip('"')
+ break
+ return out
+
+
def context_window(hostport: str, key: str) -> int:
req = urllib.request.Request(
f"http://{hostport}/v1/models",
@@ -155,9 +197,23 @@ def call(hostport: str, key: str, model: str, prompt: str,
def ramp(model: str, hostport: str, key: str, prompt_tokens: int,
levels: list[int], max_tokens: int, timeout: int) -> None:
ctx = context_window(hostport, key)
+ cfg = served_config(hostport, key)
+
+ if prompt_tokens == 0:
+ # A third of the window leaves room for output and for the chat
+ # template, and is large enough to put real pressure on the cache.
+ prompt_tokens = max(2000, (ctx or 40960) // 3)
print(f"\n{'=' * 70}")
- print(f"{model} @ {hostport} {prompt_tokens:,}-token prompts"
- f" window {ctx:,}" if ctx else f"{model} @ {hostport}")
+ head = f"{model} @ {hostport} {prompt_tokens:,}-token prompts"
+ if ctx:
+ head += f" window {ctx:,}"
+ print(head)
+ if cfg:
+ print(f" util {cfg.get('gpu_memory_utilization', '?')} "
+ f"kv dtype {cfg.get('cache_dtype', '?')} "
+ f"cache {int(cfg.get('kv_cache_size_tokens', 0)):,} tokens "
+ f"vllm concurrency estimate "
+ f"{cfg.get('kv_cache_max_concurrency', '?')}x")
# vLLM rejects a request whose prompt plus max_tokens exceeds the window.
if ctx and prompt_tokens + max_tokens > ctx:
print(f" ! prompt + max_tokens ({prompt_tokens + max_tokens:,}) "
@@ -217,9 +273,24 @@ def ramp(model: str, hostport: str, key: str, prompt_tokens: int,
"fp8, or lower --max-model-len. A bigger card "
"would help here.")
else:
- verdict = ("SCHEDULER -- queued while memory sits idle. "
- "Raise --max-num-batched-tokens (default 2048). "
- "A bigger card would NOT help.")
+ # Only recommend raising the batch budget if it has not been
+ # raised. vLLM does not publish max_num_batched_tokens, so
+ # infer from whether the box is behaving like the 2048
+ # default: that caps concurrency at 2-3 regardless of load.
+ looks_default = peak_run <= 3
+ if looks_default:
+ verdict = ("SCHEDULER -- queued while memory sits idle, "
+ "and only a few requests ever run at once. "
+ "That is the signature of the 2048 default: "
+ "raise --max-num-batched-tokens. A bigger "
+ "card would NOT help.")
+ else:
+ verdict = (f"SCHEDULER -- queued while memory sits idle, "
+ f"but {peak_run} ran concurrently, so the "
+ f"batch budget is already raised. This is the "
+ f"card: prefill throughput, not memory and "
+ f"not configuration. A faster card would "
+ f"help; a larger one would not.")
print(f"\n >> sustained queue at {conc} concurrent: "
f"{peak_run} running, {peak_wait} waiting, KV {peak_kv:.1f}%")
print(f" {verdict}")
@@ -235,7 +306,11 @@ def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--model")
ap.add_argument("--all", action="store_true")
- ap.add_argument("--prompt-tokens", type=int, default=32000)
+ # 0 = derive from each model's window. A fixed size is wrong across a
+ # fleet: 32k exceeds the 8B's entire 40,960 window, so --all used to fail
+ # on it every time with a 400 rather than measuring anything.
+ ap.add_argument("--prompt-tokens", type=int, default=0,
+ help="0 (default) = one third of each model's window")
ap.add_argument("--max-tokens", type=int, default=100)
ap.add_argument("--levels", default="2,4,8,16,24,32,48")
ap.add_argument("--timeout", type=int, default=900)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org