You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "qingchanghan (via GitHub)" <gi...@apache.org> on 2023/01/31 03:44:41 UTC

[GitHub] [tvm] qingchanghan opened a new pull request, #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

qingchanghan opened a new pull request, #13879:
URL: https://github.com/apache/tvm/pull/13879

   **Description**
   
   This PR adds a test script to combine CUTLASS and Ansor, which can use TensorCore kernels by CUTLASS and keep op fusion and automatic tuning of Ansor.
   
   **Modifications**
   
   1. Add a test script to show how to combine CUTLASS and Ansor.
   2. Add the `other_targets` parameter for Ansor's `extract_tasks` function, to pass cutlass target to `call_all_topi_funcs`.
   
   **Performance**
   
   * Bert-large
   * A10, CUDA 11.8
   * FP16
   * Input shape: (8, 128)
   
   Latency(ms)
   
   | Ansor (n=3000)   | CUTLASS+TOPI | CUTLASS+Ansor (n=3000) | Meta-Schedule (n=3000) |
   | ------- | ------------ | ------------- | ------------- |
   | 55.8870 | 20.2297      | 17.2543       | 19.2774       |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] qingchanghan commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "qingchanghan (via GitHub)" <gi...@apache.org>.
qingchanghan commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1410204275

   Please help review. @masahi @merrymercy @Laurawly @jwfromm @comaniac @tqchen @vinx13 @Hzfengsy @zhiics 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] masahi commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "masahi (via GitHub)" <gi...@apache.org>.
masahi commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1412705147

   >  BTW, I found that there are some errors when calling conv2d test in test_cutlass.py
   
   Is that an accuracy error? I'm aware that `assert_allclose` in `test_cutlass.py` can fail depending on CUDA version / card etc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] qingchanghan commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "qingchanghan (via GitHub)" <gi...@apache.org>.
qingchanghan commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1413288596

   > It would be great if you can improve on it (e.g., removing TODO I included) and send it to `main`.
   
   OK. I'll try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] tvm-bot commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "tvm-bot (via GitHub)" <gi...@apache.org>.
tvm-bot commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1409710011

   <!---bot-comment-->
   
   Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
   
   <!--bot-comment-ccs-start-->
    * No users to tag found in teams: `cutlass`, `ansor` <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
   
   <sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] junrushao commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "junrushao (via GitHub)" <gi...@apache.org>.
junrushao commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1409850946

   This is interesting and extremely helpful! Just curious - have you tried out CUTLASS + MetaSchedule?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] masahi commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "masahi (via GitHub)" <gi...@apache.org>.
masahi commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1413077300

   ok residual block fusion seems to be broken. The "residual" tensor is supposed to be passed as a different tensor than the input tensor to conv2d. So there should be 4 inputs, data, weight, bias, residual_tensor (which is the same as data). 
   
   But now we are getting only 3 inputs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] qingchanghan commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "qingchanghan (via GitHub)" <gi...@apache.org>.
qingchanghan commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1413274645

   > BTW, we are going to make further improvements to Meta-schedule with TensorCore recently, beginning with #13891
   
   This is cool! I will also learn Meta-schedule and keep an eye on it. Looking forward to further improvements!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] masahi commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "masahi (via GitHub)" <gi...@apache.org>.
masahi commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1413090119

   @qingchanghan Here is my fix for the broken residual block fusion test submitted to a different repo https://github.com/tlc-pack/relax/pull/380/commits/4911aa142e82db128679b6b08b553cd4b96e1ace. It would be great if you can improve on it (e.g., removing TODO I included) and send it to `main`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] qingchanghan commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "qingchanghan (via GitHub)" <gi...@apache.org>.
qingchanghan commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1410173834

   > This is interesting and extremely helpful! Just curious - have you tried out CUTLASS + MetaSchedule?
   
   Thank you for your advice! I tried CUTLASS + MetaSchedule and completed the test results. The performance diff between CUTLASS+Ansor and CUTLASS+MetaSchedule is negligible. I guess that the time ratio of non-MatMul operations is low in bert model and most tasks are optimized by CUTLASS.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] qingchanghan commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "qingchanghan (via GitHub)" <gi...@apache.org>.
qingchanghan commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1412028093

   > Please merge your test into `test_cutlass.py` (while disabling the tuning test by default), and remove code duplications.
   
   Thanks for your review! The merge is done. BTW, I found that there are some errors when calling conv2d test in `test_cutlass.py`. I guess it is the compatibility problem introduced by tvm update. I will try to fix them later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] masahi merged pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "masahi (via GitHub)" <gi...@apache.org>.
masahi merged PR #13879:
URL: https://github.com/apache/tvm/pull/13879


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] Hzfengsy commented on pull request #13879: [CUTLASS][Ansor] Combine CUTLASS and Ansor

Posted by "Hzfengsy (via GitHub)" <gi...@apache.org>.
Hzfengsy commented on PR #13879:
URL: https://github.com/apache/tvm/pull/13879#issuecomment-1412097267

   You could set a follow-up PR to fix that issue, and we can get it in quickly. 
   
   BTW, we are going to make further improvements to Meta-schedule with TensorCore recently, beginning with https://github.com/apache/tvm/pull/13891


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org