You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "shreemaan-abhishek (via GitHub)" <gi...@apache.org> on 2026/06/17 06:05:28 UTC
[PR] feat: add log_format_extra to enrich the default logger format [apisix]
shreemaan-abhishek opened a new pull request, #13568:
URL: https://github.com/apache/apisix/pull/13568
### Description
Setting `log_format` on a logger plugin (via `plugin_metadata` or plugin `conf`) **replaces** the default rich log entry with a flat custom format. The custom format only substitutes scalar `$variables`, so it silently drops every default field that has no scalar-variable equivalent: the full request/response header maps, the parsed querystring, the composed request URL, the APISIX version, and the computed `apisix_latency`.
Customers often set `log_format` just to add or rename one field, and unknowingly lose all of those.
This PR adds **`log_format_extra`**: an additive log format that keeps the default rich log entry intact and only overlays the user-defined fields on top. It is ignored when `log_format` is set (which keeps the replace semantics).
It also exposes a new core variable **`upstream_unresolved_host`** — the upstream host *before* DNS resolution (the configured domain or IP of the picked node). The default log only carries `upstream` (`$upstream_addr`, the resolved `ip:port`), so this variable lets users log the original hostname via `log_format_extra`.
### Example
```json
"plugin_metadata": {
"log_format_extra": {
"upstream_host": "$upstream_unresolved_host"
}
}
```
The log keeps every default field (`request`, `response`, `server`, `route_id`, ...) and gains `upstream_host`.
### Changes
- `apisix/utils/log-util.lua`: `get_log_entry` builds the default log then overlays `log_format_extra` (http subsystem only, since the default log is http-only).
- `apisix/core/ctx.lua`: register the `upstream_unresolved_host` variable.
- `apisix/balancer.lua`: stash the pre-DNS host on `ctx` when picking a server (single-node and balancer paths).
- `apisix/plugins/loki-logger.lua`, `apisix/plugins/file-logger.lua`: accept `log_format_extra` in conf and metadata schemas.
- Docs: `loki-logger.md`, `apisix-variable.md`.
- Tests: enrich scenario in `t/plugin/file-logger.t` (self-contained) and `t/plugin/loki-logger.t`.
### Checklist
- [x] Tests added
- [x] Docs updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add log_format_extra to enrich the default logger format [apisix]
Posted by "nic-6443 (via GitHub)" <gi...@apache.org>.
nic-6443 commented on code in PR #13568:
URL: https://github.com/apache/apisix/pull/13568#discussion_r3432783873
##########
apisix/core/ctx.lua:
##########
@@ -244,6 +244,8 @@ do
route_name = true,
service_id = true,
service_name = true,
+ -- the upstream host before DNS resolution (configured domain/host)
+ upstream_unresolved_host = true,
Review Comment:
I suggest putting this variable into nginx_tpl.lua so that it can also be used in the access log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add log_format_extra to enrich the default logger format [apisix]
Posted by "membphis (via GitHub)" <gi...@apache.org>.
membphis commented on PR #13568:
URL: https://github.com/apache/apisix/pull/13568#issuecomment-4727791811
[P2] `log_format_extra` is silently dropped by Google Cloud Logging and Splunk HEC in the default-format path.
This PR adds `log_format_extra` to every logger schema and documents it as an additive field set, but not every logger sends the enriched entry directly. In particular, `google-cloud-logging` and `splunk-hec-logging` call `log_util.get_log_entry(...)` and then rebuild the provider-specific default payload from a fixed list of fields.
Because `log_format_extra` does not set `customized = true`, these plugins still take their default mapping path:
- `google-cloud-logging` only writes fixed fields such as `route_id` and `service_id` into `jsonPayload` for the default entry.
- `splunk-hec-logging` rebuilds `event` from fixed request/response fields.
So a user can configure, for example, `log_format_extra = { upstream_host = "$upstream_unresolved_host" }`, pass schema validation, and still never see `upstream_host` in Google Cloud Logging or Splunk HEC output.
Please either merge extra fields into the final provider-specific payload for these plugins, or do not advertise/support `log_format_extra` for plugins where the default output path drops it. It would also be good to add coverage for at least one provider-specific remapping logger so this does not regress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add log_format_extra to enrich the default logger format [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek merged PR #13568:
URL: https://github.com/apache/apisix/pull/13568
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add log_format_extra to enrich the default logger format [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13568:
URL: https://github.com/apache/apisix/pull/13568#discussion_r3432965359
##########
apisix/core/ctx.lua:
##########
@@ -244,6 +244,8 @@ do
route_name = true,
service_id = true,
service_name = true,
+ -- the upstream host before DNS resolution (configured domain/host)
+ upstream_unresolved_host = true,
Review Comment:
Good idea, done in the latest commit (db37b1f82).
Declared `set $upstream_unresolved_host '';` in `ngx_tpl.lua` and routed the balancer writes through `ctx.var` (added it to `ngx_var_names`, removed it from `apisix_var_names`), mirroring how `$llm_content_risk_level` is wired. So it is now a real nginx variable usable directly in the access log format, in addition to the APISIX logger plugins.
Added a `t/core/ctx.t` case that proxies to a domain upstream and asserts the pre-DNS host shows up in the nginx access log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: add log_format_extra to enrich the default logger format [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on PR #13568:
URL: https://github.com/apache/apisix/pull/13568#issuecomment-4737304129
Good catch, fixed in the latest commit.
`google-cloud-logging` and `splunk-hec-logging` were the only two loggers that rebuild a fixed provider payload on the default (`not customized`) path (I swept all 18 loggers to confirm). `get_log_entry` now returns the resolved `log_format_extra` table as a third value, and both plugins merge it into their provider payload (`jsonPayload` for GCP, `event` for Splunk) without clobbering existing fixed fields. So `log_format_extra = { upstream_host = "$upstream_unresolved_host" }` now actually shows up in their output.
Added a Splunk test exercising the provider default-path so it cannot regress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org