You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "MasterJH5574 (via GitHub)" <gi...@apache.org> on 2024/03/15 21:39:36 UTC
[PR] [Builtin] Sliding window and sink support for PagedKVCache [tvm]
MasterJH5574 opened a new pull request, #16729:
URL: https://github.com/apache/tvm/pull/16729
This PR supports sliding window attention and attention sink for PagedKVCache, so that PagedKVCache can back models such as Mistral.
Meanwhile, this PR removes the "Attention" function (without fused-qkv) from AttentionKVCache interface, given its usage is now completely covered by the "AttentionWithFusedQKV" function. Considering the cost of maintenance, we decide to remove it for now. When in the future there is the need of this function, we will add it back.
This PR also unifies the global function names of the PagedKVCache with the KVState introduced earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Builtin] Sliding window and sink support for PagedKVCache [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen merged PR #16729:
URL: https://github.com/apache/tvm/pull/16729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org