You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by "yzh119 (via GitHub)" <gi...@apache.org> on 2023/05/09 21:42:32 UTC
[GitHub] [tvm] yzh119 opened a new pull request, #14814: [Unity][Relax] gelu-tanh operator
yzh119 opened a new pull request, #14814:
URL: https://github.com/apache/tvm/pull/14814
# Motivation
The gelu operator has different implementations, in [PyTorch](https://pytorch.org/docs/stable/generated/torch.nn.GELU.html), they are distinguished by `approximate='none'` or `approximate='tanh'`. Currently, Relax only implements the case where `approximate="none"` which depends on erf function, this PR implements the gelu operator approximated by tanh. In hugging face transformers, the gelu with tanh operator is called [gelu_new](https://github.com/huggingface/transformers/blob/366a8ca09e8dd92dfb1956c8be3118b5a2b13639/src/transformers/activations.py#L49) and was used in many models.
## Difference with erf gelu
Actually, I only observed a very slight numerical difference between the two implementations, so I'm still not sure whether to include this variant or not.
Below is a figure showing the numerical difference between the two, and the script:

```python
import numpy as np
import torch
import torch.nn.functional as F
import matplotlib.pyplot as plt
x = np.linspace(-6, 6, 200)
x = torch.from_numpy(x).float()
y_none = F.gelu(x, approximate='none')
y_tanh = F.gelu(x, approximate='tanh')
plt.plot(x.numpy(), (y_tanh - y_none).numpy(), label='difference')
plt.legend()
plt.show()
```
The largest difference is 5e-4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] vinx13 commented on pull request #14814: [Unity][Relax] gelu-tanh operator
Posted by "vinx13 (via GitHub)" <gi...@apache.org>.
vinx13 commented on PR #14814:
URL: https://github.com/apache/tvm/pull/14814#issuecomment-1549336821
please fix lint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tvm-bot commented on pull request #14814: [Unity][Relax] gelu-tanh operator
Posted by "tvm-bot (via GitHub)" <gi...@apache.org>.
tvm-bot commented on PR #14814:
URL: https://github.com/apache/tvm/pull/14814#issuecomment-1540928961
<!---bot-comment-->
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
<!--bot-comment-ccs-start-->
* cc @junrushao, @quic-sanirudh <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
<sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tvm-bot commented on pull request #14814: [Unity][Relax] gelu-tanh operator
Posted by "tvm-bot (via GitHub)" <gi...@apache.org>.
tvm-bot commented on PR #14814:
URL: https://github.com/apache/tvm/pull/14814#issuecomment-1540928975
<!---bot-comment-->
Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
<!--bot-comment-ccs-start-->
* cc @junrushao, @quic-sanirudh <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
<sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen merged pull request #14814: [Unity][Relax] gelu-tanh operator
Posted by "tqchen (via GitHub)" <gi...@apache.org>.
tqchen merged PR #14814:
URL: https://github.com/apache/tvm/pull/14814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org