[GitHub] [incubator-mxnet] hendrycks commented on a change in pull request #14449: GELU Operator

[GitBox <gi...@apache.org>]

hendrycks commented on a change in pull request #14449: GELU Operator
URL: https://github.com/apache/incubator-mxnet/pull/14449#discussion_r266189998
 
 

 ##########
 File path: tests/python/unittest/test_operator.py
 ##########
 @@ -862,6 +862,39 @@ def fselu_grad(grad, x, y):
         check_symbolic_backward(y, [xa], [np.ones(shape)], [ga], rtol=rtol, atol=atol, dtype=dtype)
 
 
+@with_seed()
+def test_gelu():
+    CUBE_CONSTANT = 0.044715
+    TWO_OVER_PI = 0.7978845608028654
+    def g(x):
+        return TWO_OVER_PI * (x + CUBE_CONSTANT * np.power(x, 3))
 
 Review comment:
   Often x*x*x is cheaper than x ** 3. Also, if one takes inspiration from https://en.wikipedia.org/wiki/Horner's_method , which is often more numerically stable, then g(x) could be
   
   `return TWO_OVER_PI * x * (1 + CUBE_CONSTANT * x * x)`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services