You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tooling.apache.org by "andrewmusselman (via GitHub)" <gi...@apache.org> on 2026/09/12 23:49:16 UTC
[I] Routes should use the hosted_vllm/ provider prefix, not openai/ (tooling-llmao)
andrewmusselman opened a new issue, #34:
URL: https://github.com/apache/tooling-llmao/issues/34
Every self-hosted route is created with `litellm_params.model` set to
`openai/<served-model-name>`:
```json
"key": "openai/qwen3.8-27b",
"litellm_provider": "openai"
```
LiteLLM's vLLM docs say the correct prefix for an OpenAI-compatible vLLM
server is **`hosted_vllm/`**. `openai/` tells LiteLLM it is talking to OpenAI
itself.
> Provider Route on LiteLLM: `hosted_vllm/` (for OpenAI compatible server),
> `vllm/` ([DEPRECATED] for vLLM sdk usage)
## Why it matters — two problems it likely explains
**`reasoning_effort` is validated against the wrong vocabulary.** Claude Code
sends `reasoning_effort: high`, which OpenAI accepts and Qwen3.8-27B does
not:
```
400 OpenAIException - {"error":{"message":"Unexpected reasoning effort high.
Supported types are xhigh (default), medium, and low."}}
```
LiteLLM passed it straight through because its capability map for the
`openai` provider says the parameter is supported. `drop_params: true` is
already on and did not help, for the same reason — there was nothing LiteLLM
believed needed dropping.
The docs show `reasoning_effort` as a first-class `hosted_vllm/` feature, so
the provider has its own handling for it.
**And backend errors may translate badly.** We have seen a vLLM 422 arrive as
HTTP 200 with a body of literal `null`, and a 400 arrive as
`litellm.InternalServerError` with a truncated message. A provider that knows
it is speaking to vLLM should map these more faithfully than one assuming
OpenAI's error shapes.
Worth testing rather than assuming, but both symptoms are consistent with the
wrong provider.
## Where to change it
**`llmao/litellm_client.py`** — wherever the `/model/new` payload is built.
`litellm_params.model` currently gets `openai/` prefixed to the served model
name; it should be `hosted_vllm/`.
This is route-creation code rather than config, so existing routes are
unaffected until they are recreated. Worth deleting and re-pushing after the
change, or the fleet ends up with a mix.
**`model_list.yaml.example`** — each entry has
`litellm_params.model: openai/<name>`. That value is copied into the route at
registration, so it needs the same change.
**`litellm.yaml.example`** if any sample route carries the prefix.
**`docs/manual-fleet-notes.md`** should record why, since `openai/` is the
intuitive-looking choice and someone will otherwise change it back.
## How to verify before committing
Create a parallel route and compare, without touching the live one:
```bash
curl -sS -X POST http://127.0.0.1:4000/model/new \
-H "Authorization: Bearer $MK" -H 'Content-Type: application/json' \
-d '{"model_name":"qwen-hv-test","litellm_params":{"model":"hosted_vllm/qwen3.8-27b","api_base":"http://103.196.86.105:15602/v1","api_key":"'"$VLLM_API_KEY"'"}}'
curl -sS http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer $MK" -H 'Content-Type: application/json' \
-d '{"model":"qwen-hv-test","messages":[{"role":"user","content":"Say OK"}],"reasoning_effort":"high","max_tokens":20}'
```
If that succeeds where the `openai/` route 400s, the prefix is the fix.
Also worth re-testing through it: tool calling, the Anthropic `/v1/messages`
translation, vision with a base64 data URL, and an error case. Remove the
test route afterwards with `/model/delete`.
## Note
The docs also list `/rerank`, `/embeddings` and `/audio/transcriptions` as
supported on `hosted_vllm/`, plus video input via the unified files message.
None are in scope now, but they are only reachable under the correct prefix —
so this closes off capabilities as well as causing the bugs above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
Re: [I] Routes should use the hosted_vllm/ provider prefix, not openai/ (tooling-llmao)
Posted by "andrewmusselman (via GitHub)" <gi...@apache.org>.
andrewmusselman commented on issue #34:
URL: https://github.com/apache/tooling-llmao/issues/34#issuecomment-5649642768
Another reason to switch:
LiteLLM is routing to vLLM's /v1/responses, not /v1/chat/completions:
File ".../vllm/entrypoints/serve/utils/api_utils.py", line 48, in create_responses
POST /v1/responses
And it's sending tool_choice: {"type": "auto"} — the Responses-API object form — where vLLM's implementation expects the bare string "auto". Hence fourteen validation errors, all the same thing.
This is the openai/ prefix again. LiteLLM treats the backend as OpenAI, which supports the Responses API, so it uses that shape. vLLM implements Responses only partially. With hosted_vllm/ it should use chat/completions — the endpoint the docs list, and the one our direct curl tests worked against.
That explains why /v1/messages tool calling worked in our test but fails in Claude Code: our curl went to chat/completions; Claude Code's agent loop is taking the Responses path.
Repro, in Claude Code, making a web search tool call against gemma:
```bash
❯ is there a slack command to create a github issue for a project
● Web Search("slack command to create github issue official app")
Error: API Error: 400 litellm.BadRequestError: OpenAIException - {"error":{"message":"14 validation errors:\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', \"literal['none','auto','required']\"), 'msg': \"Input should be 'none', 'auto' or 'required'\", 'input': {'type': 'auto'}, 'ctx': {'expected': \"'none', 'auto' or 'required'\"}}\n {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'mode'), 'msg': 'Field required', 'input': {'type': 'auto'}}\n {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'tools'), 'msg': 'Field required', 'input': {'type': 'auto'}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'type'), 'msg': \"Input should be 'allowed_tools'\", 'input': 'auto', 'ctx': {'expected': \"'allowed_tools'\"}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceTypes', 'type'), 'msg': \"Input should be 'file_search', 'web_search_preview', 'computer', 'computer_use_previe
w', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation' or 'code_interpreter'\", 'input': 'auto', 'ctx': {'expected': \"'file_search', 'web_search_preview', 'computer', 'computer_use_preview', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation' or 'code_interpreter'\"}}\n {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceFunction', 'name'), 'msg': 'Field required', 'input': {'type': 'auto'}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceFunction', 'type'), 'msg': \"Input should be 'function'\", 'input': 'auto', 'ctx': {'expected': \"'function'\"}}\n {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceMcp', 'server_label'), 'msg': 'Field required', 'input': {'type': 'auto'}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceMcp', 'type'), 'msg': \"Input should be 'mcp'\", 'input': 'auto', 'ctx': {'expected': \"'mcp'\"}}\n {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoic
eCustom', 'name'), 'msg': 'Field required', 'input': {'type': 'auto'}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceCustom', 'type'), 'msg': \"Input should be 'custom'\", 'input': 'auto', 'ctx': {'expected': \"'custom'\"}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceSpecificProgrammaticToolCallingParam', 'type'), 'msg': \"Input should be 'programmatic_tool_calling'\", 'input': 'auto', 'ctx': {'expected': \"'programmatic_tool_calling'\"}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceApplyPatch', 'type'), 'msg': \"Input should be 'apply_patch'\", 'input': 'auto', 'ctx': {'expected': \"'apply_patch'\"}}\n {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceShell', 'type'), 'msg': \"Input should be 'shell'\", 'input': 'auto', 'ctx': {'expected': \"'shell'\"}}\n\n File \"/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/serve/utils/api_utils.py\", line 48, in create_responses\n POST
/v1/responses [{'type': 'literal_error', 'loc': ('body', 'tool_choice', \"literal['none','auto','required']\"), 'msg': \"Input should be 'none', 'auto' or 'required'\", 'input': {'type': 'auto'}, 'ctx': {'expected': \"'none', 'auto' or 'required'\"}}, {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'mode'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'tools'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceAllowed', 'type'), 'msg': \"Input should be 'allowed_tools'\", 'input': 'auto', 'ctx': {'expected': \"'allowed_tools'\"}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceTypes', 'type'), 'msg': \"Input should be 'file_search', 'web_search_preview', 'computer', 'computer_use_preview', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation'put': 'auto', 'ctx': {'expected':
\"'file_search','web_search_preview', 'computer', 'computer_use_preview', 'computer_use', 'web_search_preview_2025_03_11', 'image_generation' or 'code_interpreter'\"}}, {'type': 'missing', 'loc': ChoiceFunction', 'name'), 'msg': 'Field required','input': {'type': 'auto'}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceFunction', 'type'), 'msg': \"Input should be 'function'\", 'input': 'auto', 'ctx': {'e'type': 'missing', 'loc': ('body', 'tool_choice','ToolChoiceMcp', 'server_label'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceMcp', 'type'), 'msg': \"It': 'auto', 'ctx': {'expected': \"'mcp'\"}}, {'type':'missing', 'loc': ('body', 'tool_choice', 'ToolChoiceCustom', 'name'), 'msg': 'Field required', 'input': {'type': 'auto'}}, {'type':'literal_error', 'loc': ('body', 'tool_choice', 'Tog': \"Input should be 'custom'\", 'input': 'auto','ctx': {'expected': \"'custom'\"}}, {'type': 'l
iteral_error', 'loc': ('body', 'tool_choice', 'ToolChoiceSpecificProgrammaticToolCallingParam', ' be 'programmatic_tool_calling'\", 'input': 'auto','ctx': {'expected': \"'programmatic_tool_calling'\"}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceApplyPatc'type'), 'msg': \"Input should be 'apply_patch'\", ected': \"'apply_patch'\"}}, {'type': 'literal_error', 'loc': ('body', 'tool_choice', 'ToolChoiceShell', 'type'), 'msg': \"Input should be 'shell'\", 'input': 'auto', 'ctx': {'expected': \"'shell'\"}}]","type":"Bad Request","param":"body.ceived Model Group=gemma4-26b
Available Model Group Fallbacks=None
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
Re: [I] Routes should use the hosted_vllm/ provider prefix, not openai/ (tooling-llmao)
Posted by "andrewmusselman (via GitHub)" <gi...@apache.org>.
andrewmusselman commented on issue #34:
URL: https://github.com/apache/tooling-llmao/issues/34#issuecomment-5735041750
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org
Re: [I] Routes should use the hosted_vllm/ provider prefix, not openai/ (tooling-llmao)
Posted by "andrewmusselman (via GitHub)" <gi...@apache.org>.
andrewmusselman closed issue #34: Routes should use the hosted_vllm/ provider prefix, not openai/
URL: https://github.com/apache/tooling-llmao/issues/34
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@tooling.apache.org
For additional commands, e-mail: commits-help@tooling.apache.org