You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "nic-6443 (via GitHub)" <gi...@apache.org> on 2026/05/21 06:06:28 UTC
[PR] perf(ai): reuse raw request body when unchanged [apisix]
nic-6443 opened a new pull request, #13406:
URL: https://github.com/apache/apisix/pull/13406
This avoids re-encoding the ai-proxy request body when the outgoing body is unchanged. If protocol conversion, provider preparation, model options, request body overrides, or provider cleanup changes the body, the existing encoded table path is still used.
It also covers ai-proxy-multi through the shared provider build path and keeps retry behavior from falling back to the original raw body after an earlier attempt has rewritten the parsed body. The raw body is read lazily in the unchanged-body branch and cached on the request context, so retries do not read it again.
The sidecar LLM request path used by ai-request-rewrite is explicitly marked as generated, so it continues to send the JSON-encoded LLM request instead of reusing the original client body.
Tests run:
```
luacheck apisix/plugins/ai-proxy/base.lua apisix/plugins/ai-providers/base.lua apisix/plugins/ai-protocols/openai-chat.lua apisix/plugins/ai-protocols/bedrock-converse.lua apisix/plugins/ai-protocols/passthrough.lua apisix/plugins/ai-request-rewrite.lua
perl -I. -c t/plugin/ai-proxy-request-body-override.t
git diff --check
prove -I. t/plugin/ai-proxy-request-body-override.t
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(ai): reuse raw request body when unchanged [apisix]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #13406:
URL: https://github.com/apache/apisix/pull/13406#discussion_r3279669464
##########
t/plugin/ai-proxy-request-body-override.t:
##########
@@ -174,7 +174,129 @@ __DATA__
-=== TEST 3: llm_options: openai provider maps max_tokens to max_completion_tokens
+=== TEST 3a: ai-proxy forwards the original body when it is not rewritten
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "uri": "/chat",
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai-compatible",
+ "auth": { "header": { "Authorization": "Bearer t" } },
+ "override": {
+ "endpoint": "http://localhost:6732"
+ },
+ "ssl_verify": false
+ }
+ }
+ }]]
+ )
+ if code >= 300 then ngx.status = code; return end
+
+ local raw = '{ "messages" : [ { "role" : "user", "content" : "hi" } ], "temperature" : 0.7 }'
+ local http = require("resty.http").new()
+ local res = assert(http:request_uri("http://127.0.0.1:" .. ngx.var.server_port .. "/chat", {
+ method = "POST",
+ body = raw,
+ headers = { ["Content-Type"] = "application/json" },
+ }))
+ local cjson = require("cjson.safe")
+ local body = cjson.decode(res.body)
+ ngx.say(body.choices[1].message.content == raw and "same body" or "body changed")
+ }
+ }
+--- response_body
+same body
+
+
+
+=== TEST 4b: ai-proxy-multi forwards the original body when it is not rewritten
Review Comment:
```
=== 4a
=== 4b
```
At this point, Lint will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(ai): reuse raw request body when unchanged [apisix]
Posted by "nic-6443 (via GitHub)" <gi...@apache.org>.
nic-6443 merged PR #13406:
URL: https://github.com/apache/apisix/pull/13406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(ai): reuse raw request body when unchanged [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13406:
URL: https://github.com/apache/apisix/pull/13406#discussion_r3279485316
##########
t/plugin/ai-proxy-request-body-override.t:
##########
@@ -174,7 +174,129 @@ __DATA__
-=== TEST 3: llm_options: openai provider maps max_tokens to max_completion_tokens
+=== TEST 3a: ai-proxy forwards the original body when it is not rewritten
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "uri": "/chat",
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai-compatible",
+ "auth": { "header": { "Authorization": "Bearer t" } },
+ "override": {
+ "endpoint": "http://localhost:6732"
+ },
+ "ssl_verify": false
+ }
+ }
+ }]]
+ )
+ if code >= 300 then ngx.status = code; return end
+
+ local raw = '{ "messages" : [ { "role" : "user", "content" : "hi" } ], "temperature" : 0.7 }'
+ local http = require("resty.http").new()
+ local res = assert(http:request_uri("http://127.0.0.1:" .. ngx.var.server_port .. "/chat", {
+ method = "POST",
+ body = raw,
+ headers = { ["Content-Type"] = "application/json" },
+ }))
+ local cjson = require("cjson.safe")
+ local body = cjson.decode(res.body)
+ ngx.say(body.choices[1].message.content == raw and "same body" or "body changed")
+ }
+ }
+--- response_body
+same body
+
+
+
+=== TEST 4b: ai-proxy-multi forwards the original body when it is not rewritten
Review Comment:
I think lint will fail here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(ai): reuse raw request body when unchanged [apisix]
Posted by "AlinsRan (via GitHub)" <gi...@apache.org>.
AlinsRan commented on code in PR #13406:
URL: https://github.com/apache/apisix/pull/13406#discussion_r3279669464
##########
t/plugin/ai-proxy-request-body-override.t:
##########
@@ -174,7 +174,129 @@ __DATA__
-=== TEST 3: llm_options: openai provider maps max_tokens to max_completion_tokens
+=== TEST 3a: ai-proxy forwards the original body when it is not rewritten
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "uri": "/chat",
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai-compatible",
+ "auth": { "header": { "Authorization": "Bearer t" } },
+ "override": {
+ "endpoint": "http://localhost:6732"
+ },
+ "ssl_verify": false
+ }
+ }
+ }]]
+ )
+ if code >= 300 then ngx.status = code; return end
+
+ local raw = '{ "messages" : [ { "role" : "user", "content" : "hi" } ], "temperature" : 0.7 }'
+ local http = require("resty.http").new()
+ local res = assert(http:request_uri("http://127.0.0.1:" .. ngx.var.server_port .. "/chat", {
+ method = "POST",
+ body = raw,
+ headers = { ["Content-Type"] = "application/json" },
+ }))
+ local cjson = require("cjson.safe")
+ local body = cjson.decode(res.body)
+ ngx.say(body.choices[1].message.content == raw and "same body" or "body changed")
+ }
+ }
+--- response_body
+same body
+
+
+
+=== TEST 4b: ai-proxy-multi forwards the original body when it is not rewritten
Review Comment:
```
===TEST 4a
===TEST 4b
```
At this point, Lint will fail.
##########
t/plugin/ai-proxy-request-body-override.t:
##########
@@ -174,7 +174,129 @@ __DATA__
-=== TEST 3: llm_options: openai provider maps max_tokens to max_completion_tokens
+=== TEST 3a: ai-proxy forwards the original body when it is not rewritten
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "uri": "/chat",
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai-compatible",
+ "auth": { "header": { "Authorization": "Bearer t" } },
+ "override": {
+ "endpoint": "http://localhost:6732"
+ },
+ "ssl_verify": false
+ }
+ }
+ }]]
+ )
+ if code >= 300 then ngx.status = code; return end
+
+ local raw = '{ "messages" : [ { "role" : "user", "content" : "hi" } ], "temperature" : 0.7 }'
+ local http = require("resty.http").new()
+ local res = assert(http:request_uri("http://127.0.0.1:" .. ngx.var.server_port .. "/chat", {
+ method = "POST",
+ body = raw,
+ headers = { ["Content-Type"] = "application/json" },
+ }))
+ local cjson = require("cjson.safe")
+ local body = cjson.decode(res.body)
+ ngx.say(body.choices[1].message.content == raw and "same body" or "body changed")
+ }
+ }
+--- response_body
+same body
+
+
+
+=== TEST 4b: ai-proxy-multi forwards the original body when it is not rewritten
Review Comment:
```
===TEST 4a
...
===TEST 4b
```
At this point, Lint will fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] perf(ai): reuse raw request body when unchanged [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13406:
URL: https://github.com/apache/apisix/pull/13406#discussion_r3279491384
##########
t/plugin/ai-proxy-request-body-override.t:
##########
@@ -174,7 +174,129 @@ __DATA__
-=== TEST 3: llm_options: openai provider maps max_tokens to max_completion_tokens
+=== TEST 3a: ai-proxy forwards the original body when it is not rewritten
+--- config
+ location /t {
+ content_by_lua_block {
+ local t = require("lib.test_admin").test
+ local code = t('/apisix/admin/routes/1',
+ ngx.HTTP_PUT,
+ [[{
+ "uri": "/chat",
+ "plugins": {
+ "ai-proxy": {
+ "provider": "openai-compatible",
+ "auth": { "header": { "Authorization": "Bearer t" } },
+ "override": {
+ "endpoint": "http://localhost:6732"
+ },
+ "ssl_verify": false
+ }
+ }
+ }]]
+ )
+ if code >= 300 then ngx.status = code; return end
+
+ local raw = '{ "messages" : [ { "role" : "user", "content" : "hi" } ], "temperature" : 0.7 }'
+ local http = require("resty.http").new()
+ local res = assert(http:request_uri("http://127.0.0.1:" .. ngx.var.server_port .. "/chat", {
+ method = "POST",
+ body = raw,
+ headers = { ["Content-Type"] = "application/json" },
+ }))
+ local cjson = require("cjson.safe")
+ local body = cjson.decode(res.body)
+ ngx.say(body.choices[1].message.content == raw and "same body" or "body changed")
+ }
+ }
+--- response_body
+same body
+
+
+
+=== TEST 4b: ai-proxy-multi forwards the original body when it is not rewritten
Review Comment:
strange that lint didn't fail here, I remember doing something similar and lint failed, lol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org