You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2017/11/01 20:25:06 UTC
[GitHub] DickJC123 opened a new issue #8509: test_operator_gpu.py:test_batchnorm_with_type robustness
DickJC123 opened a new issue #8509: test_operator_gpu.py:test_batchnorm_with_type robustness
URL: https://github.com/apache/incubator-mxnet/issues/8509
I find that in adding the line...
```
np.random.seed(1422267103)
```
...to the beginning of test_batchnorm_with_type, I get the error:
```
AssertionError:
Items are not equal:
Error 1.218629 exceeds tolerance rtol=0.100000, atol=0.100000. Location of maximum error:(0,), a=0.240414, b=0.105713
a: array([ 0.24041367, -2213.54394531], dtype=float32)
b: array([ 0.10571289, -2214. ], dtype=float16)
FAIL
======================================================================
FAIL: test_operator_gpu.test_batchnorm_with_type
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/dcarter/mxnet/tests/python/gpu/test_operator_gpu.py", line 316, in test_batchnorm_with_type
check_consistency(sym, ctx_list_v2_2D)
File "/home/dcarter/mxnet_dev/dgx/mxnet/python/mxnet/test_utils.py", line 1337, in check_consistency
raise e
AssertionError:
Items are not equal:
Error 1.218629 exceeds tolerance rtol=0.100000, atol=0.100000. Location of maximum error:(0,), a=0.240414, b=0.105713
a: array([ 0.24041367, -2213.54394531], dtype=float32)
b: array([ 0.10571289, -2214. ], dtype=float16)
```
The out-of-spec result is computed in float16 on the CPU. While perhaps @cjolivier01 or @antinucleon are looking at this, please note repetitions in some of the syms tested. Instead of the current:
```
# V2, 2D
sym = mx.sym.BatchNorm(name='norm', fix_gamma=False, cudnn_off=True)
check_consistency(sym, ctx_list_v2_2D)
sym = mx.sym.BatchNorm(name='norm', fix_gamma=False, cudnn_off=True)
check_consistency(sym, ctx_list_v2_2D)
sym = mx.sym.BatchNorm(name='norm', fix_gamma=True, cudnn_off=True)
check_consistency(sym, ctx_list_v2_2D)
sym = mx.sym.BatchNorm(name='norm', fix_gamma=True, cudnn_off=True)
check_consistency(sym, ctx_list_v2_2D)
```
Shouldn't it be:
```
# V2, 2D
sym = mx.sym.BatchNorm(name='norm', fix_gamma=False, cudnn_off=False)
check_consistency(sym, ctx_list_v2_2D)
sym = mx.sym.BatchNorm(name='norm', fix_gamma=False, cudnn_off=True)
check_consistency(sym, ctx_list_v2_2D)
sym = mx.sym.BatchNorm(name='norm', fix_gamma=True, cudnn_off=False)
check_consistency(sym, ctx_list_v2_2D)
sym = mx.sym.BatchNorm(name='norm', fix_gamma=True, cudnn_off=True)
check_consistency(sym, ctx_list_v2_2D)
```
If you end up revisiting the numerical stability of the calculation, @mkolod recommends Welford's algorithm, as he describes in https://gist.github.com/mkolod/f15727a381c537a87f4368f4ecf39022 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services