[I] Add prompt caching (tooling-gofannon)

["andrewmusselman (via GitHub)" <gi...@apache.org>]

andrewmusselman opened a new issue, #45:
URL: https://github.com/apache/tooling-gofannon/issues/45

   The key facts for designing the caching patch:
   
   * messages is passed straight into kwargs and on to litellm.acompletion(**kwargs) untouched (line ~98). So if a caller puts cache_control blocks into message content, litellm already forwards them to Bedrock — no call_llm change is strictly required for caching to work, if the message content is in the structured block form.
   * But every agent currently passes messages=[{"role":"user","content": "<big string>"}] — a plain string, which can't carry cache_control.
   * litellm.drop_params = True is set, which matters: unknown params get dropped, but cache_control inside content blocks is not a top-level param, so it survives.
   
   So the cleanest, lowest-risk design is: add an optional cache_system_prefix parameter to call_llm that, when provided, restructures the first message's content into a cacheable block + the volatile remainder. This confines all the Bedrock cache-block knowledge to one place and lets agents opt in with a single kwarg, rather than every agent learning the block format.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tooling.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add prompt caching (tooling-gofannon)

["andrewmusselman (via GitHub)" <gi...@apache.org>]

andrewmusselman commented on issue #45:
URL: https://github.com/apache/tooling-gofannon/issues/45#issuecomment-4724419525

   Done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tooling.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add prompt caching (tooling-gofannon)

["andrewmusselman (via GitHub)" <gi...@apache.org>]

andrewmusselman closed issue #45: Add prompt caching
URL: https://github.com/apache/tooling-gofannon/issues/45


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@tooling.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org