You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "Lunderberg (via GitHub)" <gi...@apache.org> on 2023/12/29 21:34:12 UTC
[PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Lunderberg opened a new pull request, #16315:
URL: https://github.com/apache/tvm/pull/16315
If `R.matmul(x, R.take(weights, indices))` occurs, with `R.take` selecting along the output feature dimension, it can be rearranged to `R.take(R.matmul(x, weights), indices)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on PR #16315:
URL: https://github.com/apache/tvm/pull/16315#issuecomment-1885955786
what is the usecase of this pass? Seems such reordering would increase the amount of compute
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on PR #16315:
URL: https://github.com/apache/tvm/pull/16315#issuecomment-1904519267
Rebased onto `main`, now that the main branch includes `unity`. No other changes made, just wanting to avoid stale CI before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on PR #16315:
URL: https://github.com/apache/tvm/pull/16315#issuecomment-1901175971
Sounds good, and thank you! I should have led with this being an optional pass to be used as needed, not intended as part of a default pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg commented on PR #16315:
URL: https://github.com/apache/tvm/pull/16315#issuecomment-1887437915
This pass would be part of an optimization pipeline that could be used for batched LoRA models. In this usage, the `R.take(weights, indices)` would select which LoRA should be used for each prompt in the batch. By rearranging the matmul, the resulting `R.matmul(x, weights)` could be combined with `R.matmul(x, base_weights)` using `CombineParallelMatmul`.
While this individual pass does increase the total amount of compute, the overall pipeline should (dependent on model size, number of loras, and batch size) improve performance by changing from three matmuls (one large matmul of the base weights and two small matmuls with the LoRA components) to two matmuls (one large matmul of `concat(base_weights, lora_a)` and one small matmul with `lora_b`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on PR #16315:
URL: https://github.com/apache/tvm/pull/16315#issuecomment-1887476314
I feel for LoRA maybe the best approach we need some specialized kernels. cc @yzh119 to see if you have more suggestions based on your experience
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen commented on PR #16315:
URL: https://github.com/apache/tvm/pull/16315#issuecomment-1900569985
i agree as long as it is optional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Unity][Transform] Implement relax.transform.ReorderTakeAfterMatmul [tvm]
Posted by "Lunderberg (via GitHub)" <gi...@apache.org>.
Lunderberg merged PR #16315:
URL: https://github.com/apache/tvm/pull/16315
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org