You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by "shreemaan-abhishek (via GitHub)" <gi...@apache.org> on 2026/07/03 07:11:04 UTC
[PR] fix(ai-cache): preserve non-text content in exact key and L2 bypass [apisix]
shreemaan-abhishek opened a new pull request, #13654:
URL: https://github.com/apache/apisix/pull/13654
### Description
`ai-protocols`' `get_messages()` flattens structured message content (images, tool calls) down to plain text. That silently broke `ai-cache` fidelity at **both** cache layers:
1. **L1 exact key** — `key.fingerprint` hashes the `get_messages` output. After flattening, a `[{text}, {image_url}]` prompt and a plain-text prompt carrying the same text collapse to the identical string, so they collide on one L1 key and the multimodal request is wrongly served the text-only response.
2. **L2 semantic bypass** — `window_has_nontext` only tripped on `type(content) == "table"`, which no longer exists after flattening, so multimodal prompts stopped bypassing the semantic layer and could hit a same-text vector.
### Fix
- `key.lua` — fold the raw `body.messages` into the L1 fingerprint so a text+image prompt stays distinct from a text-only one. (`params` already preserves every other body field verbatim, so `input`-based protocols were never affected; only `messages` lost fidelity.)
- `semantic.lua` — replace `window_has_nontext` with `body_has_nontext`, which scans the **raw** body and bypasses L2 whenever any block is non-text. It also guards non-table message items and `content` shaped as a single block object.
### Why it looked flaky
`t/plugin/ai-cache-semantic.t` TEST 64 asserts a multimodal prompt is a MISS. Whether the collision surfaces depends on TEST 63's async log-phase L2 write-back racing TEST 64's read (`--- wait: 0.5`), so it reproduces intermittently. It reproduces deterministically against a local RediSearch.
### Tests
- `t/plugin/ai-cache-semantic.t` — TEST 64 (end-to-end MISS) now passes; added TEST 66 unit-testing the malformed-input handling. Full file green (215 tests), luacheck clean.
Checklist:
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change (N/A: internal bugfix, no behavior/schema change)
- [x] I have verified that this change is backward compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ai-cache): preserve non-text content in exact key and L2 bypass [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek merged PR #13654:
URL: https://github.com/apache/apisix/pull/13654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ai-cache): preserve non-text content in exact key and L2 bypass [apisix]
Posted by "shreemaan-abhishek (via GitHub)" <gi...@apache.org>.
shreemaan-abhishek commented on code in PR #13654:
URL: https://github.com/apache/apisix/pull/13654#discussion_r3518959059
##########
apisix/plugins/ai-cache/key.lua:
##########
@@ -103,6 +103,10 @@ end
-- ONLY place request-determining data lives; scope() below is pure isolation.
function _M.fingerprint(ctx, body)
local repr = build_repr(ctx, body, client_messages(ctx, body))
+ -- client_messages() (get_messages) flattens structured content to plain text,
+ -- so an exact fingerprint of it alone would let a text+image prompt collide
+ -- with a text-only one. Fold the raw messages in to keep them distinct.
+ repr.raw_messages = body.messages
return hex_digest(core.json.canonical_encode(repr))
Review Comment:
can be handled in a later PR, we need to merge this PR soon to fix the failing CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(ai-cache): preserve non-text content in exact key and L2 bypass [apisix]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13654:
URL: https://github.com/apache/apisix/pull/13654#discussion_r3518859325
##########
apisix/plugins/ai-cache/key.lua:
##########
@@ -103,6 +103,10 @@ end
-- ONLY place request-determining data lives; scope() below is pure isolation.
function _M.fingerprint(ctx, body)
local repr = build_repr(ctx, body, client_messages(ctx, body))
+ -- client_messages() (get_messages) flattens structured content to plain text,
+ -- so an exact fingerprint of it alone would let a text+image prompt collide
+ -- with a text-only one. Fold the raw messages in to keep them distinct.
+ repr.raw_messages = body.messages
return hex_digest(core.json.canonical_encode(repr))
Review Comment:
Adding `repr.raw_messages = body.messages` unconditionally changes the L1 fingerprint for *all* `messages`-based requests, which will invalidate existing cache entries even for plain-text prompts. If the intent is only to prevent collisions for prompts with non-text/tool-call state, consider folding raw messages into the fingerprint only when such non-text state is actually present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org