You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/07/03 18:15:03 UTC
[GitHub] roywei removed a comment on issue #8044: Fix the Test
test_operator_gpu.test_batchnorm_training
roywei removed a comment on issue #8044: Fix the Test test_operator_gpu.test_batchnorm_training
URL: https://github.com/apache/incubator-mxnet/issues/8044#issuecomment-401506153
it may due to cuDNN implementation in BatchNorm.
changed
```
numeric_eps=1e-3
atol=1e-3
rtol=0.16
```
running test
```
MXNET_TEST_SEED=612977881 nosetests -s --verbose test_operator_gpu.py:test_batchnorm_training
```
give error:
```
======================================================================
FAIL: test_operator_gpu.test_batchnorm_training
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
self.test(*self.arg)
File "/usr/local/lib/python3.5/dist-packages/nose/util.py", line 620, in newfunc
return func(*arg, **kw)
File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/common.py", line 157, in test_new
orig_test(*args, **kwargs)
File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_operator.py", line 1513, in test_batchnorm_training
check_batchnorm_training(stype)
File "/home/ubuntu/incubator-mxnet/tests/python/gpu/../unittest/test_operator.py", line 1465, in check_batchnorm_training
check_numeric_gradient(test, in_location, mean_std, numeric_eps=1e-3, rtol=0.16, atol=1e-3)
File "/usr/local/lib/python3.5/dist-packages/mxnet/test_utils.py", line 914, in check_numeric_gradient
("NUMERICAL_%s"%name, "BACKWARD_%s"%name))
File "/usr/local/lib/python3.5/dist-packages/mxnet/test_utils.py", line 493, in assert_almost_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
Error 1.417775 exceeds tolerance rtol=0.160000, atol=0.001000. Location of maximum error:(0,), a=-0.001431, b=-0.000010
NUMERICAL_batchnorm22_gamma: array([-0.00143051, 0.8763224 , -0.47454238], dtype=float32)
BACKWARD_batchnorm22_gamma: array([-0.00001038, 0.87640995, -0.47491556], dtype=float32)
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=797368411 to reproduce.
common: WARNING: *** test-level seed set: all "@with_seed()" tests run deterministically ***
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=612977881 to reproduce.
--------------------- >> end captured logging << ---------------------
```
This test is disabled for both cpu and gpu, we should enable cpu case at least.
#11396 didn't have this issue because, [check consistency](https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/test_utils.py#L1207) is not using training during forward execution. while check_numeric_gradients use `is_train=True`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services