You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "joshua-j-hong (via GitHub)" <gi...@apache.org> on 2025/05/07 19:02:05 UTC

[PR] KV Cache Per Layer Sliding Window [tvm]

joshua-j-hong opened a new pull request, #17928:
URL: https://github.com/apache/tvm/pull/17928

   Adds per layer sliding window functionality to the KV Cache. Currently there are some major bugs that should be resolved before merging. 
   
   1. Throughput slows down significantly, even before the sliding window length is reached
   2. Correctness issues past the sliding window length, potentially due to page indexing or some other KV cache issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [KVCache] Per Layer Sliding Window [tvm]

Posted by "joshua-j-hong (via GitHub)" <gi...@apache.org>.
joshua-j-hong commented on PR #17928:
URL: https://github.com/apache/tvm/pull/17928#issuecomment-2993759160

   Conflicts and tests are fixed. Current plans are for a future change that will add optional parameters to the KVCache, some of which will be for layer sliding window. This will ensure that no hardcoding of values will be needed and the KVCache will be backwards compatible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [KVCache] Per Layer Sliding Window [tvm]

Posted by "MasterJH5574 (via GitHub)" <gi...@apache.org>.
MasterJH5574 commented on PR #17928:
URL: https://github.com/apache/tvm/pull/17928#issuecomment-2963848335

   Just see some conflicts with upstream. Likely we need to do a rebase.  Related changes are the recent FFI refactor and a namespace rename from `relax_vm` to `vm`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [KVCache] Per Layer Sliding Window [tvm]

Posted by "MasterJH5574 (via GitHub)" <gi...@apache.org>.
MasterJH5574 merged PR #17928:
URL: https://github.com/apache/tvm/pull/17928


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [KVCache] Per Layer Sliding Window [tvm]

Posted by "joshua-j-hong (via GitHub)" <gi...@apache.org>.
joshua-j-hong commented on PR #17928:
URL: https://github.com/apache/tvm/pull/17928#issuecomment-2864024276

   With some further testing and investigation, there is an additional MLC-LLM/TVM bug that is related to excessive prefilling (even without the per-layer sliding window changes outlined here) that is related to inference slowdown.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org