[GitHub] anirudhacharya commented on issue #11509: Hanging flaky test test_operator.test_norm @ Python 3: GPU Win

[GitBox <gi...@apache.org>]

anirudhacharya commented on issue #11509: Hanging flaky test test_operator.test_norm @ Python 3: GPU Win
URL: https://github.com/apache/incubator-mxnet/issues/11509#issuecomment-409726747
 
 
   Update on this issue - 
   
   I ran this test case ~5k times for different scenarios on a Linux GPU machine (EC2 p2.8x large instance). Currently the reasons for test failure are - 
   1. ``check_numeric_gradient`` has precision issues when run with np.float16 and np.float32. I have updated set of atol and rtol values that can fix this. But the new atol and rtol values still have to be verified with repeated runs of the test.
   2. The bigger bug is that the test intermittently hangs while performing the ``check_numeric_gradient``. ``check_symbolic_backward`` and ``check_symbolic_forward`` works correctly and consistently.
   
   The above two issues are only reproducible when run against CPU context. When run against GPU the test neither hangs nor does it have accuracy issues.
   
   I need to check the same with a Windows instance.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services