Re: [I] bug: ai-request-rewrite plugin returns LLM output directly instead of forwarding to upstream [apisix]

["shreemaan-abhishek (via GitHub)" <gi...@apache.org>]

shreemaan-abhishek commented on issue #13447:
URL: https://github.com/apache/apisix/issues/13447#issuecomment-4609122499

   Thanks for the detailed report @gardnerhan. I looked into this and can confirm it is a real regression. Your diagnosis is correct.
   
   **Root cause**
   
   The behavior regressed in **APISIX 3.14.0** via #12530. That PR moved `ai-request-rewrite` onto the shared ai-driver response path:
   
   - The original implementation (3.13.x and earlier) called the LLM in the `access` phase, extracted `choices[1].message.content`, and then called `ngx.req.set_body_data()` so the rewritten body continued on to the route's upstream. This is the behavior described in the [plugin docs](https://apisix.apache.org/docs/apisix/plugins/ai-request-rewrite/).
   - Since 3.14.0, the plugin's `access` phase hands the request to `ai_driver:request()` (`apisix/plugins/ai-drivers/openai-base.lua`), which ends by calling `plugin.lua_response_filter()`. That function runs the plugin's `lua_body_filter` hook (which wraps the LLM content as `{"data": ...}`) and then sends it directly to the client with `ngx.print()` + `ngx.exit()`. The configured upstream is never contacted.
   
   The `{"data":"{\"name\":\"John\",\"processed\":true}"}` response you observed is exactly that wrapper from `parse_llm_response()` in `apisix/plugins/ai-request-rewrite.lua`, not your upstream's response.
   
   The reason CI never caught this: the existing test (`t/plugin/ai-request-rewrite.t`, TEST 8) asserts on `response_data.data`, which happens to pass under both the correct behavior (the test upstream echoes the request body in a `data` field) and the broken behavior (the plugin's wrapper is also named `data`).
   
   **Affected versions**
   
   3.14.0, 3.14.1, 3.15.0, 3.16.0
   
   **Status**
   
   This is already fixed on `master` by #13170 (the AI proxy architecture refactor). The plugin now extracts the LLM content and calls `ngx.req.set_body_data()` again, so the rewritten body is forwarded to the upstream as documented. The fix is not in any published release yet; it will ship in the next release after 3.16.0.
   
   **Workaround until then**
   
   Downgrade to 3.13.x if this plugin's documented behavior is critical for you, or build from `master`.
   
   I'd also suggest a follow-up to harden the test so it can distinguish "upstream received the rewritten body" from "plugin returned the LLM output directly" (e.g. by asserting on an upstream-only echo field). I can open a PR for that if maintainers agree.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org