You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "Lunderberg (via GitHub)" <gi...@apache.org> on 2024/03/06 16:05:19 UTC
[PR] [Bugfix][SLM] Produce well-formed Relax for nn.modules.KVCache [tvm]
Lunderberg opened a new pull request, #16684:
URL: https://github.com/apache/tvm/pull/16684
Prior to this commit, the `nn.modules.KVCache` implementations used `R.call_packed(...)` to call the `"vm.builtin.attention_*"` functions. Since `nn.Module` emits all relax functions within a `relax.DataflowBlock`, where impure expressions are forbidden, this is ill-formed.
This commit updates the implementations in `nn.modules.KVCache` to use `R.call_pure_packed` instead of `R.call_packed`. This assertation that the callee is pure allows the call to occur within a `relax.DataflowBlock`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bugfix][SLM] Produce well-formed Relax for nn.modules.KVCache [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on PR #16684:
URL: https://github.com/apache/tvm/pull/16684#issuecomment-1981548246
That makes sense with the migration. I've been assuming that the translation layer from SLM to Relax would still exist with the unified KV cache, and so it would need a corresponding change either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bugfix][SLM] Produce well-formed Relax for nn.modules.KVCache [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on PR #16684:
URL: https://github.com/apache/tvm/pull/16684#issuecomment-1981540370
one note: we are migrating into the latest unified kv so likely we might want to remove the legacy kv builtin soon in favor of the new kv
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bugfix][SLM] Produce well-formed Relax for nn.modules.KVCache [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen merged PR #16684:
URL: https://github.com/apache/tvm/pull/16684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Bugfix][SLM] Produce well-formed Relax for nn.modules.KVCache [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on PR #16684:
URL: https://github.com/apache/tvm/pull/16684#issuecomment-1987301161
All unit tests now passing, ready for review/merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org