You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "AlinsRan (via GitHub)" <gi...@apache.org> on 2026/06/05 01:04:10 UTC
[PR] feat(ai-proxy): add built-in nginx variables for LLM observability [apisix]
AlinsRan opened a new pull request, #13477:
URL: https://github.com/apache/apisix/pull/13477
## Summary
Add 8 built-in nginx variables that `ai-proxy` sets automatically on every request. These variables expose LLM request/response metadata in the nginx access log and all logger plugins without any additional configuration.
### New variables
| Variable | Type | Description |
|---|---|---|
| `$llm_total_tokens` | integer | Total tokens (prompt + completion) |
| `$llm_stream` | boolean | `true` if request is streaming |
| `$llm_has_tool_calls` | boolean | `true` if response contains tool calls |
| `$llm_tool_count` | integer | Number of tools in the upstream request body |
| `$llm_end_user_id` | string | End-user identifier from request body |
| `$llm_cache_read_input_tokens` | integer | Prompt tokens served from provider cache |
| `$llm_cache_creation_input_tokens` | integer | Prompt tokens written to provider cache |
| `$llm_reasoning_tokens` | integer | Reasoning tokens (OpenAI o1/o3, Responses API) |
### Provider mapping for cache/reasoning tokens
| Variable | OpenAI Chat | OpenAI Responses | Anthropic | DeepSeek |
|---|---|---|---|---|
| `llm_cache_read_input_tokens` | `prompt_tokens_details.cached_tokens` | `input_tokens_details.cached_tokens` | `cache_read_input_tokens` | `prompt_cache_hit_tokens` |
| `llm_cache_creation_input_tokens` | — | — | `cache_creation_input_tokens` | — |
| `llm_reasoning_tokens` | `completion_tokens_details.reasoning_tokens` | `output_tokens_details.reasoning_tokens` | — | — |
End-user ID extraction precedence: `safety_identifier` > `user` (OpenAI/compatible) or `metadata.user_id` (Anthropic Messages).
### Example access log usage
```nginx
log_format main '$llm_model $llm_total_tokens $llm_stream $llm_has_tool_calls $llm_end_user_id';
```
### Also includes
- Optional `on_event` callback parameter added to `parse_streaming_response` for per-event processing (e.g., streaming tool call detection).
- `extract_usage` updated across all three protocol adapters to extract cache and reasoning token counts from provider responses.
## Files changed
| File | Change |
|---|---|
| `apisix/cli/ngx_tpl.lua` | Add 8 `set $llm_*` variable definitions |
| `apisix/core/ctx.lua` | Register 8 variables in `ngx_var_names` |
| `apisix/plugins/ai-protocols/openai-chat.lua` | Extract cache/reasoning tokens from `prompt_tokens_details` / `completion_tokens_details` |
| `apisix/plugins/ai-protocols/openai-responses.lua` | Extract cache/reasoning tokens from `input_tokens_details` / `output_tokens_details` |
| `apisix/plugins/ai-protocols/anthropic-messages.lua` | Extract cache tokens from Anthropic usage fields |
| `apisix/plugins/ai-providers/base.lua` | Set new variables from usage; add `on_event` callback |
| `apisix/plugins/ai-proxy/base.lua` | Compute `llm_stream`, `llm_tool_count`, `llm_end_user_id`, `llm_has_tool_calls`, `llm_total_tokens` |
| `t/APISIX.pm` | Add `set` directives + extend `log_format main` |
| `t/plugin/ai-proxy3.t` | 6 test cases (TEST 7–12) covering all new variables |
## Test plan
```bash
prove -I. -r t/plugin/ai-proxy3.t
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): add built-in nginx variables for LLM observability [apisix]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan merged PR #13477:
URL: https://github.com/apache/apisix/pull/13477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): add built-in nginx variables for LLM observability [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13477:
URL: https://github.com/apache/apisix/pull/13477#discussion_r3386743357
##########
apisix/plugins/ai-protocols/anthropic-messages.lua:
##########
@@ -66,6 +66,14 @@ function _M.parse_sse_event(event, ctx, state)
end
return { type = "skip" }
+ elseif event.type == "content_block_start" then
+ local data = core.json.decode(event.data, { null_as_nil = true })
+ if data and type(data.content_block) == "table"
+ and data.content_block.type == "tool_use" then
+ return { type = "skip", has_tool_call = true }
+ end
+ return { type = "skip" }
+
Review Comment:
this change doesn't seem related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): add built-in nginx variables for LLM observability [apisix]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #13477:
URL: https://github.com/apache/apisix/pull/13477#discussion_r3386819211
##########
apisix/plugins/ai-protocols/anthropic-messages.lua:
##########
@@ -66,6 +66,14 @@ function _M.parse_sse_event(event, ctx, state)
end
return { type = "skip" }
+ elseif event.type == "content_block_start" then
+ local data = core.json.decode(event.data, { null_as_nil = true })
+ if data and type(data.content_block) == "table"
+ and data.content_block.type == "tool_use" then
+ return { type = "skip", has_tool_call = true }
+ end
+ return { type = "skip" }
+
Review Comment:
It is used to determine whether LLM has called the tools
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org