You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by gs...@apache.org on 2026/10/11 22:42:15 UTC
(tooling-llmao) branch main updated: models: add a BF16 Qwen3 8B for cards the FP8 build cannot load on
This is an automated email from the ASF dual-hosted git repository.
gstein pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tooling-llmao.git
The following commit(s) were added to refs/heads/main by this push:
new e26cc53 models: add a BF16 Qwen3 8B for cards the FP8 build cannot load on
e26cc53 is described below
commit e26cc53f955079b94dab53e7653cb614b5d88eee
Author: Greg Stein <gs...@gmail.com>
AuthorDate: Sun Oct 11 17:41:22 2026 -0500
models: add a BF16 Qwen3 8B for cards the FP8 build cannot load on
Qwen's FP8 checkpoints are block-quantized, which vLLM runs through
DeepGEMM. DeepGEMM only handles SM90 and SM100, so on a GB10 (compute
capability 12.1) the engine dies during load with "Unknown SF
transformation". qwen3-8b-bf16 serves the unquantized weights, which
never reach that code.
It is a separate model_name: callers ask for it by that name and it does
not pool with the qwen3-8b boxes. No pricing yet; nothing has been
measured on the GB10.
Co-authored-by: Claude Opus 5.5 <no...@anthropic.com>
---
models.yaml | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 65 insertions(+)
diff --git a/models.yaml b/models.yaml
index 5e11287..8e5f81d 100644
--- a/models.yaml
+++ b/models.yaml
@@ -278,3 +278,68 @@ models:
clocks, so that is the card's real rate rather than a limit worth
chasing. The advantage over the 27B is concurrency, not per-request
speed.
+
+ # The same model as qwen3-8b, from the unquantized weights. For cards the
+ # FP8 build cannot load on.
+ #
+ # Qwen's FP8 checkpoints are block-quantized (weight_block_size [128, 128]),
+ # which vLLM runs through DeepGEMM, and DeepGEMM only knows SM90 (Hopper)
+ # and SM100 (datacenter Blackwell). On a GB10 (compute capability 12.1) the
+ # engine dies while loading with "Unknown SF transformation" and nothing
+ # more. BF16 never reaches that code.
+ #
+ # A separate model_name, so callers ask for it by this name and it does not
+ # pool with the qwen3-8b boxes.
+ - model_name: qwen3-8b-bf16
+ litellm_params:
+ model: hosted_vllm/qwen3-8b-bf16
+ # As for qwen3-8b: unusable with reasoning on by default.
+ chat_template_kwargs:
+ enable_thinking: false
+ # No pricing: nothing has been measured on the GB10, so usage prices at
+ # zero until it is.
+ model_info:
+ vllm:
+ model: Qwen/Qwen3-8B
+ source:
+ kind: hf
+ # Derived, not measured: 8.19B parameters at two bytes is 15.3 GiB.
+ # Rounded up so the fit check fails closed.
+ vram_gb: 17
+ # 15.3G of weights in the HF repo.
+ disk_gb: 20
+ # Architectural ceiling; see qwen3-8b.
+ max_model_len: 40960
+ args:
+ - --dtype
+ - bfloat16
+ # The weights are BF16 but the KV cache can still be FP8. This is a
+ # cast in the attention path, not DeepGEMM.
+ - --kv-cache-dtype
+ - fp8
+ - --enable-prefix-caching
+ - --enable-chunked-prefill
+ # Carried over from qwen3-8b, not tuned for this box.
+ - --max-num-batched-tokens
+ - "4096"
+ - --reasoning-parser
+ - qwen3
+ - --enable-auto-tool-choice
+ - --tool-call-parser
+ - hermes
+ display_name: "Qwen3 8B BF16 (self-hosted, fast)"
+ provider: self-host
+ context_window: 40960
+ license: Apache-2.0
+ openness: open-weight
+ weights_distribution: "Qwen/Qwen3-8B (HF, BF16) ยท vLLM"
+ training_data_provenance: "undisclosed (Alibaba)"
+ provenance_record: absent
+ self_hosted: true
+ modality: text
+ supports_thinking: true
+ thinks_by_default: true
+ notes: >-
+ Qwen3 8B from the unquantized weights, for hardware the FP8 build
+ does not load on. Same model and same behaviour as qwen3-8b; send
+ enable_thinking:false to keep it cheap. Not yet measured.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org