[GitHub] luobao-intel edited a comment on issue #12377: Flaky test: test_mkldnn.test_activation

[GitBox <gi...@apache.org>]

luobao-intel edited a comment on issue #12377: Flaky test: test_mkldnn.test_activation
URL: https://github.com/apache/incubator-mxnet/issues/12377#issuecomment-416861592
 
 
   This test is to validate the activation calculation in mkldnn by checking the gradient compared to the theano.gradient.numeric_grad. However, the activation gradient calculation of code based on theano is not correct with the input closed to zero.  Thus, flaky errors occurred when there are some extremely small positive numbers in the random input vector. 
   The experiment is as follows.
   
   ## experiment 1:
   
   input data :[[1, 2], [3, 0.0001]]
   
   location:
   {'data':
   <RowSparseNDArray 2x2 @cpu(0)>, '__random_proj':
   [[0.3546685  0.8954062 ]
    [0.40476447 0.7724642 ]]
   <NDArray 2x2 @cpu(0)>}
   
   theano gradient :
   [[0.35466552 0.8954048 ]
    [0.40476322 0.39395675]]
   
   mkldnn :
   [[0.3546685  0.8954062 ]
    [0.40476447 0.7724642 ]]
   
   ## experiment 2:
   input data :[[1, -2], [-4, 0.0005]]
   
   location:
   {'data':
   <RowSparseNDArray 2x2 @cpu(0)>, '__random_proj':
   [[0.3546685  0.8954062 ]
    [0.40476447 0.7724642 ]]
   <NDArray 2x2 @cpu(0)>}
   
   theano gradient :
   [[0.35466552 0.        ]
    [0.         0.4248553 ]]
   
   mkldnn :
   [[0.3546685 0.       ]
    [0.        0.7724642]]
   
   ## analysis
   It's easy to know that the derivative  of ReLU function is :
   if x < 0, output is 0. if x > 0, output is 1.
   
   Therefore, in the check_numeric_gradient function, the gradient of executor should be equal to location if the corresponding element of input data is positive and be 0 otherwise by element-wise. 
   The gradient based on theano is apparently false when the corresponding element of input data is close to zero. 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services