You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "shreemaan-abhishek (via GitHub)" <gi...@apache.org> on 2026/08/05 05:01:27 UTC
[PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
shreemaan-abhishek opened a new pull request, #13778:
URL: https://github.com/apache/apisix/pull/13778
### Description
`ai-proxy`, `ai-proxy-multi` and `ai-request-rewrite` all send their outbound
LLM requests through `apisix/plugins/ai-transport/http.lua`. This makes that
transport prefer `ngx_http_ffi_client`, an HTTP client implemented as an nginx C
module, and keeps `lua-resty-http` as the fallback.
`ngx_http_ffi_client` exposes the same object API as `lua-resty-http`
(`new`, `set_timeout`, `connect`, `request`, `res.body_reader`,
`res:read_body`, `set_keepalive`, `close`) and does the HTTP/1.1 framing in C,
which costs roughly a third of the outbound CPU time on this path.
The module only exists when the runtime was built with it, so the transport
resolves the client once per worker and falls back when it is absent. **On a
runtime without the module nothing changes**: the same `lua-resty-http` code
runs as before.
`plugin_attr.ai-proxy.http_client` pins the choice for operators who want it
fixed either way:
```yaml
plugin_attr:
ai-proxy:
http_client: auto # auto (default), ffi, or lua-resty-http
```
One behaviour change applies to both clients: `Connection` and
`Transfer-Encoding` are now dropped from the headers forwarded to the LLM
upstream, alongside `Host`, `Content-Length` and `Accept-Encoding`. Both are
hop-by-hop headers describing the downstream connection. The client frames the
request body with `Content-Length` itself, so forwarding a client's
`Transfer-Encoding: chunked` put two conflicting framing headers on the wire,
and forwarding `Connection: close` desynchronised the upstream connection and
its keepalive pool.
#### Which issue(s) this PR fixes:
N/A
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not, please discuss on the [APISIX mailing list](https://github.com/apache/apisix/tree/master#community) first)
### Test plan
`t/plugin/ai-transport-http.t` gains four cases covering client selection: the
FFI client is used when present, the transport falls back for the rest of the
worker's life when the module's Lua half loads but the C module is not built in,
`plugin_attr` pins `lua-resty-http` over an available FFI client, and hop-by-hop
headers are dropped from the forwarded set.
Beyond the plugin tests, the client itself was exercised against a real build:
an OpenResty carrying the module, driven through the exact call sequence this
transport uses, covering a buffered request, an SSE stream read through
`body_reader`, `set_keepalive`, `close`, and the error strings `handle_error`
maps to 502/504.
`t/plugin/ai-proxy*.t` and `t/plugin/ai-transport-*.t` were run before and after
the change with identical results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on PR #13778:
URL: https://github.com/apache/apisix/pull/13778#issuecomment-5190573841
Pushed the config change: `plugin_attr.ai-proxy.http_client` now takes one of the two client names and defaults to the C client.
```yaml
plugin_attr:
ai-proxy:
http_client: ngx_http_ffi_client # or lua-resty-http
```
`auto` and `ffi` are gone. `auto` made the choice implicit in whatever the runtime happened to carry, which is the opposite of what a config knob is for.
**Coverage of the `lua-resty-http` path.** TEST 10 selects it through `plugin_attr` and asserts the request goes through the Lua client, but it does that against a stub. TEST 11 is new and drives the real thing: `http_client: lua-resty-http`, nothing stubbed, a real request to a local upstream, asserting status, `Content-Type`, body and `set_keepalive`.
**One decision worth a look.** With no `auto`, a runtime built without the module and left on the default has to do something. I kept the fallback to `lua-resty-http` and log it, rather than failing the request, because until a released runtime carries the module that is every deployment, and failing would turn this into a breaking change. The log stays at warn rather than error for the same reason: at error level every AI request on such a runtime writes an error line, and it broke the `no_error_log: [error]` blocks in the existing suite when I tried it.
If you would rather the default hard-fail when the module is missing, that is a small change, but it means these two PRs cannot merge before a runtime carrying the module ships.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek merged PR #13778:
URL: https://github.com/apache/apisix/pull/13778
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13778:
URL: https://github.com/apache/apisix/pull/13778#discussion_r3765250765
##########
apisix/plugins/ai-transport/http.lua:
##########
@@ -19,17 +19,130 @@
-- Provides HTTP client lifecycle management for AI provider requests.
local core = require("apisix.core")
-local http = require("resty.http")
local ngx_now = ngx.now
local pairs = pairs
local ipairs = ipairs
local pcall = pcall
+local require = require
local type = type
local str_lower = string.lower
local tostring = tostring
+local FFI_CLIENT = "ngx_http_ffi_client"
+local LUA_RESTY_HTTP = "lua-resty-http"
+
+-- the client name in the config is not the module name
+local CLIENT_MODULES = {
+ [FFI_CLIENT] = "resty.ngx_http_ffi_client",
+ [LUA_RESTY_HTTP] = "resty.http",
+}
+
+local attr_schema = {
+ type = "object",
+ properties = {
+ http_client = {
+ type = "string",
+ enum = {FFI_CLIENT, LUA_RESTY_HTTP},
+ default = FFI_CLIENT,
+ },
+ },
+}
+
local _M = {}
+local http_client
+local http_client_is_ffi
+
+
+--- Pick the outbound HTTP client.
+-- `plugin_attr.ai-proxy.http_client` names it: "ngx_http_ffi_client", the
+-- default, or "lua-resty-http". The first is a C client with the same object
+-- API as the second and around half its outbound CPU cost, and it exists only
+-- when the gateway runtime was built with the module.
+-- Resolved on first request, because local_conf is not readable while the
+-- module is still loading, and cached only once a client has been loaded.
+local function resolve_client()
+ if http_client then
+ return http_client
+ end
+
+ local local_conf = core.config.local_conf()
+ local attr = core.table.try_read_attr(local_conf, "plugin_attr", "ai-proxy") or {}
+
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("invalid plugin_attr.ai-proxy: ", err)
+ return nil, "invalid plugin_attr.ai-proxy: " .. err
+ end
+
+ local name = attr.http_client or FFI_CLIENT
+ local module_name = CLIENT_MODULES[name]
+
+ local mod
+ ok, mod = pcall(require, module_name)
+ if not ok or type(mod) ~= "table" then
+ core.log.error(module_name, " is not available: ", mod)
+ return nil, module_name .. " is not available: " .. tostring(mod)
+ end
+
+ http_client = mod
+ http_client_is_ffi = name == FFI_CLIENT
+
+ return http_client
+end
+
+
+--- Resolve the upstream name the way every other socket in the gateway does.
+-- Cosockets are patched (apisix/patch.lua) to run names through
+-- core.resolver, which honours dns_resolver, /etc/hosts and the search
+-- domains. The C client dials on its own and only sees nginx's `resolver`,
+-- so the name is resolved here and kept for the Host header and the SNI.
+local function resolve_upstream_host(params)
Review Comment:
Filed as api7/ngx_http_ffi_client#45.
Worth separating two things in your question. The duplication part is addressed in this PR: the resolution now lives in `apisix/utils/http.lua` rather than in the AI transport, so plugins share it instead of each carrying a copy. What a hook would fix is the remaining footgun — every caller still has to remember to call `resolve_upstream_host()` before `connect()`, and forgetting it fails only against hostnames that nginx's resolver cannot answer, which is easy to miss locally.
One design note I put in the issue: this cannot follow the same route as the trust-store fix in #43. There the value lived in ngx_lua's loc conf and was readable from C. `core.resolver` is a Lua module whose lookup yields, so C cannot call it synchronously — the hook has to sit in the Lua binding and resolve before entering the FFI boundary. That makes it cheap, no C change needed.
Three details from implementing the workaround that the hook needs to preserve, all in the issue: resolve before the keepalive pool key is derived (two names resolving to different addresses must not share a pooled connection, the same invariant #43 established for the CA), keep the original name for the `Host` header and the SNI (the binding defaults `ssl_server_name` to `opts.host`, so substituting first silently turns the SNI into an IP and breaks verification), and short-circuit IP literals.
Once it lands, `resolve_upstream_host()` and its call site come out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on PR #13778:
URL: https://github.com/apache/apisix/pull/13778#issuecomment-5188028833
Pushed a follow-up removing the hop-by-hop header change from this PR.
`ai-cache-streaming.t` TEST 37 and 38 failed in CI. It reproduces locally on this branch and not on `main`, and reverting only the blacklist hunk makes it go away, so the cause was dropping `Connection` from the forwarded headers, not the client cutover. The symptom is an `ai-proxy-multi` retry after a stream that dies before its first byte: it stops finding an instance to re-pick and returns 502 with `failed to pick new AI instance: all upstream servers tried`.
Two things are worth following up separately, neither of which belongs in a PR about which HTTP client the transport builds:
- Forwarding a downstream `Connection` header to a third-party LLM upstream is wrong per RFC 9110, and it feeds the client's keepalive decision.
- The streaming read-error path in `parse_streaming_response` returns without closing `res._httpc`, so a timed-out upstream connection is leaked. That looks like the real reason the retry behaves differently once `Connection: close` stops reaching the upstream.
This PR is now just the client selection, and the transport diff stays byte-identical with the paired PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13778:
URL: https://github.com/apache/apisix/pull/13778#discussion_r3748009598
##########
apisix/plugins/ai-transport/http.lua:
##########
@@ -19,17 +19,131 @@
-- Provides HTTP client lifecycle management for AI provider requests.
local core = require("apisix.core")
-local http = require("resty.http")
local ngx_now = ngx.now
local pairs = pairs
local ipairs = ipairs
local pcall = pcall
+local require = require
local type = type
local str_lower = string.lower
+local tonumber = tonumber
local tostring = tostring
+local FFI_CLIENT = "ngx_http_ffi_client"
+local LUA_RESTY_HTTP = "lua-resty-http"
+
+-- the client name in the config is not the module name
+local CLIENT_MODULES = {
+ [FFI_CLIENT] = "resty.ngx_http_ffi_client",
+ [LUA_RESTY_HTTP] = "resty.http",
+}
+
+local attr_schema = {
+ type = "object",
+ properties = {
+ http_client = {
+ type = "string",
+ enum = {FFI_CLIENT, LUA_RESTY_HTTP},
+ default = FFI_CLIENT,
+ },
+ },
+}
+
local _M = {}
+local http_client
+local http_client_is_ffi
+
+
+--- Pick the outbound HTTP client.
+-- `plugin_attr.ai-proxy.http_client` names it: "ngx_http_ffi_client", the
+-- default, or "lua-resty-http". The first is a C client with the same object
+-- API as the second and around half its outbound CPU cost, and it exists only
+-- when the gateway runtime was built with the module.
+-- Resolved on first request, because local_conf is not readable while the
+-- module is still loading, and cached only once a client has been loaded.
+local function resolve_client()
+ if http_client then
+ return http_client
+ end
+
+ local local_conf = core.config.local_conf()
+ local attr = core.table.try_read_attr(local_conf, "plugin_attr", "ai-proxy") or {}
+
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("invalid plugin_attr.ai-proxy: ", err)
+ return nil, "invalid plugin_attr.ai-proxy: " .. err
+ end
+
+ local name = attr.http_client or FFI_CLIENT
+ local module_name = CLIENT_MODULES[name]
+
+ local mod
+ ok, mod = pcall(require, module_name)
+ if not ok or type(mod) ~= "table" then
+ core.log.error(module_name, " is not available: ", mod)
+ return nil, module_name .. " is not available: " .. tostring(mod)
+ end
+
+ http_client = mod
+ http_client_is_ffi = name == FFI_CLIENT
+
+ return http_client
+end
Review Comment:
Good point, moved. The selection now lives in `apisix/utils/http.lua`, next to the other shared client factories (`utils/redis.lua` is the closest precedent), and `ai-transport/http.lua` is just a consumer.
The split keeps the reusable half in the shared module and the plugin-specific half out of it:
- `utils/http.lua` owns the client names, a `client_schema` fragment callers embed in their own attribute schema, loading and per-name caching, and `resolve_upstream_host()`.
- the caller owns **where the preference comes from** and passes the name to `new(name)`. So `ai-transport` reads `plugin_attr.ai-proxy.http_client`, and forward-auth or http-logger can read their own key without inheriting an AI-specific one.
`resolve_upstream_host()` is the part most worth sharing. `apisix/patch.lua` wraps cosocket `connect` so hostnames go through `core.resolver`, which honours `dns_resolver`, `/etc/hosts` and the search domains. A client that dials from C never touches a cosocket and sees only nginx's `resolver`, so without this every `localhost`-style upstream fails. Any plugin moving off lua-resty-http will hit that, so it should not be reimplemented per plugin.
I deliberately did **not** introduce a global config key. Making `http_client` gateway-wide is a user-facing decision that felt like yours to make rather than something to slip into this PR; the shared module is agnostic, so a global default can be added later without touching callers.
One option I skipped: having `new()` return a wrapper whose `connect()` resolves transparently, so callers would not need the `needs_resolve()` check. That removes a step for every future caller but adds a metatable indirection on a per-request path, so I left it explicit. Happy to switch if you prefer it.
No behaviour change; `t/plugin/ai-transport-http.t` (61 subtests) plus `ai-proxy.t`, `ai-proxy-multi.t`, `ai-request-rewrite.t` and `ai-transport-header-forwarding.t` all pass unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "nic-6443 (via GitHub)" <gi...@apache.org>.
nic-6443 commented on code in PR #13778:
URL: https://github.com/apache/apisix/pull/13778#discussion_r3747582985
##########
apisix/plugins/ai-transport/http.lua:
##########
@@ -19,17 +19,131 @@
-- Provides HTTP client lifecycle management for AI provider requests.
local core = require("apisix.core")
-local http = require("resty.http")
local ngx_now = ngx.now
local pairs = pairs
local ipairs = ipairs
local pcall = pcall
+local require = require
local type = type
local str_lower = string.lower
+local tonumber = tonumber
local tostring = tostring
+local FFI_CLIENT = "ngx_http_ffi_client"
+local LUA_RESTY_HTTP = "lua-resty-http"
+
+-- the client name in the config is not the module name
+local CLIENT_MODULES = {
+ [FFI_CLIENT] = "resty.ngx_http_ffi_client",
+ [LUA_RESTY_HTTP] = "resty.http",
+}
+
+local attr_schema = {
+ type = "object",
+ properties = {
+ http_client = {
+ type = "string",
+ enum = {FFI_CLIENT, LUA_RESTY_HTTP},
+ default = FFI_CLIENT,
+ },
+ },
+}
+
local _M = {}
+local http_client
+local http_client_is_ffi
+
+
+--- Pick the outbound HTTP client.
+-- `plugin_attr.ai-proxy.http_client` names it: "ngx_http_ffi_client", the
+-- default, or "lua-resty-http". The first is a C client with the same object
+-- API as the second and around half its outbound CPU cost, and it exists only
+-- when the gateway runtime was built with the module.
+-- Resolved on first request, because local_conf is not readable while the
+-- module is still loading, and cached only once a client has been loaded.
+local function resolve_client()
+ if http_client then
+ return http_client
+ end
+
+ local local_conf = core.config.local_conf()
+ local attr = core.table.try_read_attr(local_conf, "plugin_attr", "ai-proxy") or {}
+
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("invalid plugin_attr.ai-proxy: ", err)
+ return nil, "invalid plugin_attr.ai-proxy: " .. err
+ end
+
+ local name = attr.http_client or FFI_CLIENT
+ local module_name = CLIENT_MODULES[name]
+
+ local mod
+ ok, mod = pcall(require, module_name)
+ if not ok or type(mod) ~= "table" then
+ core.log.error(module_name, " is not available: ", mod)
+ return nil, module_name .. " is not available: " .. tostring(mod)
+ end
+
+ http_client = mod
+ http_client_is_ffi = name == FFI_CLIENT
+
+ return http_client
+end
Review Comment:
These codes should not be placed directly under the `apisix/plugins/ai-transport` module. Our plan is to gradually replace the dependencies on lua‑resty‑http across all plugins, such as forward‑auth and http‑logger. Placing the codes here will prevent them from being reused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "nic-6443 (via GitHub)" <gi...@apache.org>.
nic-6443 commented on code in PR #13778:
URL: https://github.com/apache/apisix/pull/13778#discussion_r3747591160
##########
apisix/plugins/ai-transport/http.lua:
##########
@@ -19,17 +19,130 @@
-- Provides HTTP client lifecycle management for AI provider requests.
local core = require("apisix.core")
-local http = require("resty.http")
local ngx_now = ngx.now
local pairs = pairs
local ipairs = ipairs
local pcall = pcall
+local require = require
local type = type
local str_lower = string.lower
local tostring = tostring
+local FFI_CLIENT = "ngx_http_ffi_client"
+local LUA_RESTY_HTTP = "lua-resty-http"
+
+-- the client name in the config is not the module name
+local CLIENT_MODULES = {
+ [FFI_CLIENT] = "resty.ngx_http_ffi_client",
+ [LUA_RESTY_HTTP] = "resty.http",
+}
+
+local attr_schema = {
+ type = "object",
+ properties = {
+ http_client = {
+ type = "string",
+ enum = {FFI_CLIENT, LUA_RESTY_HTTP},
+ default = FFI_CLIENT,
+ },
+ },
+}
+
local _M = {}
+local http_client
+local http_client_is_ffi
+
+
+--- Pick the outbound HTTP client.
+-- `plugin_attr.ai-proxy.http_client` names it: "ngx_http_ffi_client", the
+-- default, or "lua-resty-http". The first is a C client with the same object
+-- API as the second and around half its outbound CPU cost, and it exists only
+-- when the gateway runtime was built with the module.
+-- Resolved on first request, because local_conf is not readable while the
+-- module is still loading, and cached only once a client has been loaded.
+local function resolve_client()
+ if http_client then
+ return http_client
+ end
+
+ local local_conf = core.config.local_conf()
+ local attr = core.table.try_read_attr(local_conf, "plugin_attr", "ai-proxy") or {}
+
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("invalid plugin_attr.ai-proxy: ", err)
+ return nil, "invalid plugin_attr.ai-proxy: " .. err
+ end
+
+ local name = attr.http_client or FFI_CLIENT
+ local module_name = CLIENT_MODULES[name]
+
+ local mod
+ ok, mod = pcall(require, module_name)
+ if not ok or type(mod) ~= "table" then
+ core.log.error(module_name, " is not available: ", mod)
+ return nil, module_name .. " is not available: " .. tostring(mod)
+ end
+
+ http_client = mod
+ http_client_is_ffi = name == FFI_CLIENT
+
+ return http_client
+end
+
+
+--- Resolve the upstream name the way every other socket in the gateway does.
+-- Cosockets are patched (apisix/patch.lua) to run names through
+-- core.resolver, which honours dns_resolver, /etc/hosts and the search
+-- domains. The C client dials on its own and only sees nginx's `resolver`,
+-- so the name is resolved here and kept for the Host header and the SNI.
+local function resolve_upstream_host(params)
Review Comment:
Could ngx_http_ffi_client provide a hook for DNS resolution? Otherwise, the DNS‑resolving code would have to be duplicated across many plugins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13778:
URL: https://github.com/apache/apisix/pull/13778#discussion_r3765250765
##########
apisix/plugins/ai-transport/http.lua:
##########
@@ -19,17 +19,130 @@
-- Provides HTTP client lifecycle management for AI provider requests.
local core = require("apisix.core")
-local http = require("resty.http")
local ngx_now = ngx.now
local pairs = pairs
local ipairs = ipairs
local pcall = pcall
+local require = require
local type = type
local str_lower = string.lower
local tostring = tostring
+local FFI_CLIENT = "ngx_http_ffi_client"
+local LUA_RESTY_HTTP = "lua-resty-http"
+
+-- the client name in the config is not the module name
+local CLIENT_MODULES = {
+ [FFI_CLIENT] = "resty.ngx_http_ffi_client",
+ [LUA_RESTY_HTTP] = "resty.http",
+}
+
+local attr_schema = {
+ type = "object",
+ properties = {
+ http_client = {
+ type = "string",
+ enum = {FFI_CLIENT, LUA_RESTY_HTTP},
+ default = FFI_CLIENT,
+ },
+ },
+}
+
local _M = {}
+local http_client
+local http_client_is_ffi
+
+
+--- Pick the outbound HTTP client.
+-- `plugin_attr.ai-proxy.http_client` names it: "ngx_http_ffi_client", the
+-- default, or "lua-resty-http". The first is a C client with the same object
+-- API as the second and around half its outbound CPU cost, and it exists only
+-- when the gateway runtime was built with the module.
+-- Resolved on first request, because local_conf is not readable while the
+-- module is still loading, and cached only once a client has been loaded.
+local function resolve_client()
+ if http_client then
+ return http_client
+ end
+
+ local local_conf = core.config.local_conf()
+ local attr = core.table.try_read_attr(local_conf, "plugin_attr", "ai-proxy") or {}
+
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("invalid plugin_attr.ai-proxy: ", err)
+ return nil, "invalid plugin_attr.ai-proxy: " .. err
+ end
+
+ local name = attr.http_client or FFI_CLIENT
+ local module_name = CLIENT_MODULES[name]
+
+ local mod
+ ok, mod = pcall(require, module_name)
+ if not ok or type(mod) ~= "table" then
+ core.log.error(module_name, " is not available: ", mod)
+ return nil, module_name .. " is not available: " .. tostring(mod)
+ end
+
+ http_client = mod
+ http_client_is_ffi = name == FFI_CLIENT
+
+ return http_client
+end
+
+
+--- Resolve the upstream name the way every other socket in the gateway does.
+-- Cosockets are patched (apisix/patch.lua) to run names through
+-- core.resolver, which honours dns_resolver, /etc/hosts and the search
+-- domains. The C client dials on its own and only sees nginx's `resolver`,
+-- so the name is resolved here and kept for the Host header and the SNI.
+local function resolve_upstream_host(params)
Review Comment:
Filed as api7/ngx_http_ffi_client#45.
Worth separating two things in your question. The duplication part is addressed in this PR: the resolution now lives in `apisix/utils/http.lua` rather than in the AI transport, so plugins share it instead of each carrying a copy. What a hook would fix is the remaining footgun — every caller still has to remember to call `resolve_upstream_host()` before `connect()`, and forgetting it fails only against hostnames that nginx's resolver cannot answer, which is easy to miss locally.
One design note I put in the issue: this cannot follow the same route as the trust-store fix in #43. There the value lived in ngx_lua's loc conf and was readable from C. `core.resolver` is a Lua module whose lookup yields, so C cannot call it synchronously — the hook has to sit in the Lua binding and resolve before entering the FFI boundary. That makes it cheap, no C change needed.
Three details from implementing the workaround that the hook needs to preserve, all in the issue: resolve before the keepalive pool key is derived (two names resolving to different addresses must not share a pooled connection, the same invariant #43 established for the CA), keep the original name for the `Host` header and the SNI (the binding defaults `ssl_server_name` to `opts.host`, so substituting first silently turns the SNI into an IP and breaks verification), and short-circuit IP literals.
Once it lands, `resolve_upstream_host()` and its call site come out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13778:
URL: https://github.com/apache/apisix/pull/13778#discussion_r3728489224
##########
apisix/plugins/ai-transport/http.lua:
##########
@@ -19,17 +19,130 @@
-- Provides HTTP client lifecycle management for AI provider requests.
local core = require("apisix.core")
-local http = require("resty.http")
local ngx_now = ngx.now
local pairs = pairs
local ipairs = ipairs
local pcall = pcall
+local require = require
local type = type
local str_lower = string.lower
local tostring = tostring
+local FFI_CLIENT = "ngx_http_ffi_client"
+local LUA_RESTY_HTTP = "lua-resty-http"
+
+-- the client name in the config is not the module name
+local CLIENT_MODULES = {
+ [FFI_CLIENT] = "resty.ngx_http_ffi_client",
+ [LUA_RESTY_HTTP] = "resty.http",
+}
+
+local attr_schema = {
+ type = "object",
+ properties = {
+ http_client = {
+ type = "string",
+ enum = {FFI_CLIENT, LUA_RESTY_HTTP},
+ default = FFI_CLIENT,
+ },
+ },
+}
+
local _M = {}
+local http_client
+local http_client_is_ffi
+
+
+--- Pick the outbound HTTP client.
+-- `plugin_attr.ai-proxy.http_client` names it: "ngx_http_ffi_client", the
+-- default, or "lua-resty-http". The first is a C client with the same object
+-- API as the second and around half its outbound CPU cost, and it exists only
+-- when the gateway runtime was built with the module.
+-- Resolved on first request, because local_conf is not readable while the
+-- module is still loading, and cached only once a client has been loaded.
+local function resolve_client()
+ if http_client then
+ return http_client
+ end
+
+ local local_conf = core.config.local_conf()
+ local attr = core.table.try_read_attr(local_conf, "plugin_attr", "ai-proxy") or {}
+
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("invalid plugin_attr.ai-proxy: ", err)
+ return nil, "invalid plugin_attr.ai-proxy: " .. err
+ end
+
+ local name = attr.http_client or FFI_CLIENT
+ local module_name = CLIENT_MODULES[name]
+
+ local mod
+ ok, mod = pcall(require, module_name)
+ if not ok or type(mod) ~= "table" then
+ core.log.error(module_name, " is not available: ", mod)
+ return nil, module_name .. " is not available: " .. tostring(mod)
+ end
+
+ http_client = mod
+ http_client_is_ffi = name == FFI_CLIENT
+
+ return http_client
+end
+
+
+--- Resolve the upstream name the way every other socket in the gateway does.
+-- Cosockets are patched (apisix/patch.lua) to run names through
+-- core.resolver, which honours dns_resolver, /etc/hosts and the search
+-- domains. The C client dials on its own and only sees nginx's `resolver`,
+-- so the name is resolved here and kept for the Host header and the SNI.
+local function resolve_upstream_host(params)
Review Comment:
ngx_http_ffi_client dials from C, so it never touches the cosocket that patch.lua wraps — it only sees nginx's `resolver`, which does not read /etc/hosts or our dns_resolver config. That is why every `localhost` upstream 500'd. Resolving here puts it back on the same path as every other socket in the gateway, and we keep the name for Host and SNI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(ai-proxy): send LLM requests through ngx_http_ffi_client [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13778:
URL: https://github.com/apache/apisix/pull/13778#discussion_r3748009598
##########
apisix/plugins/ai-transport/http.lua:
##########
@@ -19,17 +19,131 @@
-- Provides HTTP client lifecycle management for AI provider requests.
local core = require("apisix.core")
-local http = require("resty.http")
local ngx_now = ngx.now
local pairs = pairs
local ipairs = ipairs
local pcall = pcall
+local require = require
local type = type
local str_lower = string.lower
+local tonumber = tonumber
local tostring = tostring
+local FFI_CLIENT = "ngx_http_ffi_client"
+local LUA_RESTY_HTTP = "lua-resty-http"
+
+-- the client name in the config is not the module name
+local CLIENT_MODULES = {
+ [FFI_CLIENT] = "resty.ngx_http_ffi_client",
+ [LUA_RESTY_HTTP] = "resty.http",
+}
+
+local attr_schema = {
+ type = "object",
+ properties = {
+ http_client = {
+ type = "string",
+ enum = {FFI_CLIENT, LUA_RESTY_HTTP},
+ default = FFI_CLIENT,
+ },
+ },
+}
+
local _M = {}
+local http_client
+local http_client_is_ffi
+
+
+--- Pick the outbound HTTP client.
+-- `plugin_attr.ai-proxy.http_client` names it: "ngx_http_ffi_client", the
+-- default, or "lua-resty-http". The first is a C client with the same object
+-- API as the second and around half its outbound CPU cost, and it exists only
+-- when the gateway runtime was built with the module.
+-- Resolved on first request, because local_conf is not readable while the
+-- module is still loading, and cached only once a client has been loaded.
+local function resolve_client()
+ if http_client then
+ return http_client
+ end
+
+ local local_conf = core.config.local_conf()
+ local attr = core.table.try_read_attr(local_conf, "plugin_attr", "ai-proxy") or {}
+
+ local ok, err = core.schema.check(attr_schema, attr)
+ if not ok then
+ core.log.error("invalid plugin_attr.ai-proxy: ", err)
+ return nil, "invalid plugin_attr.ai-proxy: " .. err
+ end
+
+ local name = attr.http_client or FFI_CLIENT
+ local module_name = CLIENT_MODULES[name]
+
+ local mod
+ ok, mod = pcall(require, module_name)
+ if not ok or type(mod) ~= "table" then
+ core.log.error(module_name, " is not available: ", mod)
+ return nil, module_name .. " is not available: " .. tostring(mod)
+ end
+
+ http_client = mod
+ http_client_is_ffi = name == FFI_CLIENT
+
+ return http_client
+end
Review Comment:
Done. The selection now lives in `apisix/utils/http.lua`, next to the other shared client factories (`utils/redis.lua` is the closest precedent), and `ai-transport/http.lua` is just a consumer.
- `utils/http.lua` owns the client names, a `client_schema` fragment callers embed in their own attribute schema, loading and per-name caching, and `resolve_upstream_host()`.
- the caller owns **where the preference comes from** and passes the name to `new(name)`. So `ai-transport` reads `plugin_attr.ai-proxy.http_client`, and forward-auth or http-logger can read their own key without inheriting an AI-specific one.
`resolve_upstream_host()` is the part most worth sharing. `apisix/patch.lua` wraps cosocket `connect` so hostnames go through `core.resolver`, which honours `dns_resolver`, `/etc/hosts` and the search domains. A client that dials from C never touches a cosocket and sees only nginx's `resolver`, so without this every `localhost`-style upstream fails. Any plugin moving off lua-resty-http will hit that, so it should not be reimplemented per plugin.
One option I skipped: having `new()` return a wrapper whose `connect()` resolves transparently, so callers would not need the `needs_resolve()` check. That removes a step for every future caller but adds a metatable indirection on a per-request path, so I left it explicit. Happy to switch if you prefer it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org