You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by gs...@apache.org on 2026/10/02 14:48:19 UTC
(tooling-llmao) 01/03: smoke: read the gateway bearer from config.yaml
This is an automated email from the ASF dual-hosted git repository.
gstein pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-llmao.git
commit 9c012ea24344e8dcf75a5c4586633a2338f50162
Author: Greg Stein <gs...@gmail.com>
AuthorDate: Fri Oct 2 09:40:59 2026 -0500
smoke: read the gateway bearer from config.yaml
llmao-smoke required LLMAO_KEY even though litellm.master_key is already
in config.yaml. Use that key unless LLMAO_KEY is set, and document loading
the override from a file so the secret stays out of shell history and ps.
Co-authored-by: Grok <gr...@x.ai>
---
bin/README.md | 12 ++++++++----
bin/llmao-smoke | 33 +++++++++++++++++++++++++++------
docs/STATUS.md | 9 +++++++--
3 files changed, 42 insertions(+), 12 deletions(-)
diff --git a/bin/README.md b/bin/README.md
index 6438e13..c13b105 100644
--- a/bin/README.md
+++ b/bin/README.md
@@ -8,8 +8,9 @@ Scripts for llmao; needs more for others but this documents model perf testing.
| `llmao-smoke` | after any change to a model, box or the gateway | ~1 min/model |
| `llmao-saturate` | when deciding whether a card is the right size | 5–30 min |
-Smoke and saturate are stdlib-only Python 3. `llmao-vllm-api-key` uses
-`llmao.vllm_api_key` and `config.yaml` unless `--fleet-key` is passed.
+`llmao-saturate` is stdlib-only Python 3. `llmao-smoke` uses PyYAML to read
+`litellm.master_key` from `config.yaml`. `llmao-vllm-api-key` uses
+`llmao.vllm_api_key` and `config.yaml` unless `--salt` is passed.
## llmao-vllm-api-key
@@ -19,7 +20,7 @@ will get from `GET /vllm/config` after that is wired. See
```bash
bin/llmao-vllm-api-key --host 203.0.113.10 --port 8001
-bin/llmao-vllm-api-key --host 203.0.113.10 --port 8001 --fleet-key sk-...
+bin/llmao-vllm-api-key --host 203.0.113.10 --port 8001 --salt sk-...
```
Paste into `vllm serve --api-key` / Supervisor on a hand-launched box.
@@ -27,12 +28,15 @@ Paste into `vllm serve --api-key` / Supervisor on a hand-launched box.
## llmao-smoke
```bash
-export LLMAO_KEY=sk-... # a PAT from My Keys
llmao-smoke # every model the gateway advertises
llmao-smoke --model qwen3-8b
+# optional override; the file holds the PAT, one line, no quotes
+read -r LLMAO_KEY < ~/.llmao-key && export LLMAO_KEY
llmao-smoke --direct # bypass LiteLLM; needs VLLM_API_KEY
```
+The gateway bearer is `litellm.master_key` in `config.yaml`. `LLMAO_KEY` overrides it. Do not put the key on the command line (`LLMAO_KEY=sk-... llmao-smoke` or `export LLMAO_KEY=sk-...`): the value is stored in shell history and is visible to other users via `ps`. `read` from a file leaves the secret out of both.
+
Six checks per model: completion, thinking-off, tool calling, vision,
sustained output, and a 25k-token prompt. Exits non-zero if anything fails,
so it works in CI.
diff --git a/bin/llmao-smoke b/bin/llmao-smoke
index 802c520..3d7cd52 100755
--- a/bin/llmao-smoke
+++ b/bin/llmao-smoke
@@ -6,9 +6,12 @@ catalog says a model accepts, and fails when the model does not behave as the
catalog says it will. Use probe.py to discover what a model can do; use this
to notice when that stops being true.
- export LLMAO_KEY=sk-... # a PAT from My Keys
- llmao-smoke # all models
+ llmao-smoke # all models; bearer is config.yaml litellm.master_key
llmao-smoke --model qwen3-8b
+ # Optional override. Read the key from a file. A literal
+ # LLMAO_KEY=sk-... on the command line is stored in shell history and
+ # is visible to other users via ps.
+ # read -r LLMAO_KEY < ~/.llmao-key && export LLMAO_KEY
llmao-smoke --direct # bypass LiteLLM; needs VLLM_API_KEY
Run this after any change to a model, a box, or the gateway. It is the
@@ -50,6 +53,7 @@ import argparse
import base64
import json
import os
+import pathlib
import struct
import sys
import time
@@ -57,7 +61,10 @@ import urllib.error
import urllib.request
import zlib
+import yaml
+
GATEWAY = "https://llm.apache.org/v1/chat/completions"
+CONFIG = pathlib.Path(__file__).resolve().parent.parent / "config.yaml"
# No endpoints are committed here. --direct discovers them from LiteLLM's
# routing table on the gateway host, or from LLMAO_DIRECT_<MODEL> env vars
@@ -386,16 +393,30 @@ def check(model: str, url: str, key: str) -> int:
return fails
+def master_key_from_config(path: pathlib.Path) -> str:
+ if not path.is_file():
+ raise SystemExit(f"Missing {path} (set LLMAO_KEY or create config.yaml)")
+ data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
+ litellm = data.get("litellm") or {}
+ key = litellm.get("master_key")
+ if not isinstance(key, str) or not key.strip() or key.strip().startswith("CHANGE_ME"):
+ raise SystemExit(f"{path}: litellm.master_key is missing or CHANGE_ME")
+ return key.strip()
+
+
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--model", action="append")
ap.add_argument("--direct", action="store_true", help="hit the boxes directly; isolates gateway problems")
+ ap.add_argument("--config", type=pathlib.Path, default=CONFIG, help="config.yaml for litellm.master_key")
args = ap.parse_args()
- var = "VLLM_API_KEY" if args.direct else "LLMAO_KEY"
- key = os.environ.get(var)
- if not key:
- sys.exit(f"export {var} first")
+ if args.direct:
+ key = os.environ.get("VLLM_API_KEY")
+ if not key:
+ sys.exit("export VLLM_API_KEY first")
+ else:
+ key = os.environ.get("LLMAO_KEY") or master_key_from_config(args.config)
endpoints = direct_endpoints() if args.direct else {}
if args.direct and not endpoints:
diff --git a/docs/STATUS.md b/docs/STATUS.md
index b50f536..aac162a 100644
--- a/docs/STATUS.md
+++ b/docs/STATUS.md
@@ -93,8 +93,13 @@ distinguishes "the parser declined" from "the model never called".
## Testing
-Two scripts in `bin/`, stdlib-only, with no endpoints committed — they
-discover from LiteLLM's routing table at runtime.
+Two scripts in `bin/`, with no endpoints committed — they discover from
+LiteLLM's routing table at runtime. `llmao-saturate` is stdlib-only.
+`llmao-smoke` reads `litellm.master_key` from `config.yaml` via PyYAML.
+`LLMAO_KEY` overrides it; load that from a file
+(`read -r LLMAO_KEY < ~/.llmao-key && export LLMAO_KEY`). A key typed on
+the command line is stored in shell history and is visible to other users
+via `ps`.
| | what it answers |
|---|---|
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org