You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "yzh119 (via GitHub)" <gi...@apache.org> on 2023/05/09 21:42:32 UTC

[GitHub] [tvm] yzh119 opened a new pull request, #14814: [Unity][Relax] gelu-tanh operator

yzh119 opened a new pull request, #14814:
URL: https://github.com/apache/tvm/pull/14814

   # Motivation
   The gelu operator has different implementations, in [PyTorch](https://pytorch.org/docs/stable/generated/torch.nn.GELU.html), they are distinguished by `approximate='none'` or `approximate='tanh'`. Currently, Relax only implements the case where `approximate="none"`  which depends on erf function, this PR implements the gelu operator approximated by tanh. In hugging face transformers, the gelu with tanh operator is called [gelu_new](https://github.com/huggingface/transformers/blob/366a8ca09e8dd92dfb1956c8be3118b5a2b13639/src/transformers/activations.py#L49) and was used in many models.
   
   ## Difference with erf gelu
   Actually, I only observed a very slight numerical difference between the two implementations, so I'm still not sure whether to include this variant or not.
   
   Below is a figure showing the numerical difference between the two, and the script:
   ![image](https://github.com/apache/tvm/assets/11773619/e002b8a5-9e0a-49ed-9e61-eebdfb7c0eba)
   
   ```python
   import numpy as np
   import torch
   import torch.nn.functional as F
   import matplotlib.pyplot as plt
   
   x = np.linspace(-6, 6, 200)
   x = torch.from_numpy(x).float()
   y_none = F.gelu(x, approximate='none')
   y_tanh = F.gelu(x, approximate='tanh')
   plt.plot(x.numpy(), (y_tanh - y_none).numpy(), label='difference')
   plt.legend()
   plt.show()
   ```
   
   The largest difference is 5e-4.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [tvm] vinx13 commented on pull request #14814: [Unity][Relax] gelu-tanh operator

Posted by "vinx13 (via GitHub)" <gi...@apache.org>.
vinx13 commented on PR #14814:
URL: https://github.com/apache/tvm/pull/14814#issuecomment-1549336821

   please fix lint


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [tvm] tvm-bot commented on pull request #14814: [Unity][Relax] gelu-tanh operator

Posted by "tvm-bot (via GitHub)" <gi...@apache.org>.
tvm-bot commented on PR #14814:
URL: https://github.com/apache/tvm/pull/14814#issuecomment-1540928961

   <!---bot-comment-->
   
   Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
   
   <!--bot-comment-ccs-start-->
    * cc @junrushao, @quic-sanirudh <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
   
   <sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [tvm] tvm-bot commented on pull request #14814: [Unity][Relax] gelu-tanh operator

Posted by "tvm-bot (via GitHub)" <gi...@apache.org>.
tvm-bot commented on PR #14814:
URL: https://github.com/apache/tvm/pull/14814#issuecomment-1540928975

   <!---bot-comment-->
   
   Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
   
   <!--bot-comment-ccs-start-->
    * cc @junrushao, @quic-sanirudh <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
   
   <sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [tvm] tqchen merged pull request #14814: [Unity][Relax] gelu-tanh operator

Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen merged PR #14814:
URL: https://github.com/apache/tvm/pull/14814


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org