You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/06/03 02:31:09 UTC
[GitHub] [incubator-mxnet] nttstar opened a new issue #18475: BatchNorm can not converge with scale=False
nttstar opened a new issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475
## Description
BatchNorm operator with ``scale=False`` can not converge.
### Error Message
No error message, but loss value and training accuracy is abnormal comparing with ``scale=True`` BatchNorm.
## To Reproduce
We can try ``https://github.com/nttstar/arcface.np`` to train arcface. Add one BatchNorm op with ``scale=False`` after final embedding layer
## What have you tried to solve it?
1. Set ``Scale=True``, it can work but with slightly worse test accuracy.
## Environment
----------Python Info----------
Version : 3.6.9
Compiler : GCC 7.3.0
Build : ('default', 'Jul 30 2019 19:07:31')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 19.3.1
Directory : /root/anaconda2/envs/py36/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version : 2.0.0
Directory : /root/anaconda2/envs/py36/lib/python3.6/site-packages/mxnet
Num GPUs : 8
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform : Linux-3.10.0-327.el7.x86_64-x86_64-with-centos-7.5.1804-Core
system : Linux
node : gpu06
release : 3.10.0-327.el7.x86_64
version : #1 SMP Thu Nov 19 22:10:57 UTC 2015
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn commented on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639377361
I try to test the batch norm with `fix_gamma=True` on GPU, the test fails on `assert_almost_equal(bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)` in [test_operator.py#L1950](https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_operator.py#L1950).
```
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, False)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, False)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, True)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, True)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 1, False)
*** Maximum errors for vector of size 2: rtol=0.01, atol=0.01
1: Error 90.669586 Location of error: (0,), a=28.82106400, b=14.64017868
2: Error 64.738541 Location of error: (1,), a=22.83149147, b=13.46625137
```
The result on CPU is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639377361
I try to test the batch norm with `fix_gamma=True`.
The result on CPU is right, but that on GPU is wrong when `fix_gamma=True, axis = 1 and cudnn_off=False`. The `bn_beta` is the only wrong value.
Here are the failure cases when `fix_gamma=True`.
operator|shape|axis|cudnn_off|output_mean_var
--|--|--|--|--
BatchNorm|(24, 2)|1|False|False
BatchNorm|(24, 2)|1|False|True
BatchNorm|(24, 3, 4)|1|False|False
BatchNorm|(24, 3, 4)|1|False|True
BatchNorm|(24, 4, 4, 4)|1|False|False
BatchNorm|(24, 4, 4, 4)|1|False|True
BatchNorm|(24, 8, 4, 4)|1|False|False
BatchNorm|(24, 8, 4, 4)|1|False|True
BatchNorm|(24, 5, 6, 4, 4)|1|False|False
BatchNorm|(24, 5, 6, 4, 4)|1|False|True
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience closed issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
sxjscience closed issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639377361
I try to test the batch norm with `fix_gamma=True` on GPU, the test fails on `assert_almost_equal(bn_beta.grad.asnumpy(), db.asnumpy(), atol=atol, rtol=rtol)` in [test_operator.py#L1950](https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_operator.py#L1950).
```
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, False)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, False)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, True)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 0, True)
(<function BatchNorm at 0x7f54854753b0>, (24, 2), 1, False)
*** Maximum errors for vector of size 2: rtol=0.01, atol=0.01
1: Error 90.669586 Location of error: (0,), a=28.82106400, b=14.64017868
2: Error 64.738541 Location of error: (1,), a=22.83149147, b=13.46625137
```
However, the result on CPU is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-641678357
Hi @nttstar , the bug of BatchNorm has been solved in #18500 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-641678357
Hi @nttstar , the bug of BatchNorm has been fixed in #18500 .
Thank you for the report!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn commented on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-641678357
Hi @nttstar , the bug of BatchNorm has been solved in #18492 .
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639377361
I try to test the batch norm with `fix_gamma=True`.
The result on CPU is right, but that on GPU is wrong when `fix_gamma=True and axis = 1`.
Here are the failure cases.
operator|shape|axis|cudnn_off|output_mean_var
--|--|--|--|--
BatchNorm|(24, 2)|1|False|False
BatchNorm|(24, 2)|1|False|True
BatchNorm|(24, 3, 4)|1|False|False
BatchNorm|(24, 3, 4)|1|False|True
BatchNorm|(24, 4, 4, 4)|1|False|False
BatchNorm|(24, 4, 4, 4)|1|False|True
BatchNorm|(24, 8, 4, 4)|1|False|False
BatchNorm|(24, 8, 4, 4)|1|False|True
BatchNorm|(24, 5, 6, 4, 4)|1|False|False
BatchNorm|(24, 5, 6, 4, 4)|1|False|True
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639377361
I try to test the batch norm with `fix_gamma=True`.
The result on CPU is right, but that on GPU is wrong when `fix_gamma=True, axis = 1 and cudnn_off=False`. The `bn_beta.grad` is the only wrong value.
Here are the failure cases when `fix_gamma=True`.
operator|shape|axis|cudnn_off|output_mean_var
--|--|--|--|--
BatchNorm|(24, 2)|1|False|False
BatchNorm|(24, 2)|1|False|True
BatchNorm|(24, 3, 4)|1|False|False
BatchNorm|(24, 3, 4)|1|False|True
BatchNorm|(24, 4, 4, 4)|1|False|False
BatchNorm|(24, 4, 4, 4)|1|False|True
BatchNorm|(24, 8, 4, 4)|1|False|False
BatchNorm|(24, 8, 4, 4)|1|False|True
BatchNorm|(24, 5, 6, 4, 4)|1|False|False
BatchNorm|(24, 5, 6, 4, 4)|1|False|True
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn commented on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639178923
Hi @nttstar , there was a bug in BatchNorm in the previous version of MXNet. The bug was reported in #18373 , and it was fixed in PR #18377 . Could you please try the latest version of MXNet?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn commented on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639359723
@sxjscience I'm sorry that I don't have any machine with GPU to check it recently.
I read the code of batch norm and its unittest.
There is a gradient check when `fix_gamma=True`:
https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_operator.py#L1777,
but no output check when `fix_gamma=True`:
https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_operator.py#L1882
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639377361
I try to test the batch norm with `fix_gamma=True`.
The result on CPU is right, but that on GPU is wrong when `fix_gamma=True and axis = 1`.
Here are the failure cases when `fix_gamma=True`.
operator|shape|axis|cudnn_off|output_mean_var
--|--|--|--|--
BatchNorm|(24, 2)|1|False|False
BatchNorm|(24, 2)|1|False|True
BatchNorm|(24, 3, 4)|1|False|False
BatchNorm|(24, 3, 4)|1|False|True
BatchNorm|(24, 4, 4, 4)|1|False|False
BatchNorm|(24, 4, 4, 4)|1|False|True
BatchNorm|(24, 8, 4, 4)|1|False|False
BatchNorm|(24, 8, 4, 4)|1|False|True
BatchNorm|(24, 5, 6, 4, 4)|1|False|False
BatchNorm|(24, 5, 6, 4, 4)|1|False|True
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn commented on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn commented on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639413806
I'm fixing the bug and I will submit a PR later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] nttstar commented on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
nttstar commented on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-641684791
@wkcn Thanks! I will check it in the next pip package release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
sxjscience commented on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639260163
@wkcn I communicated with @nttstar offline and https://github.com/apache/incubator-mxnet/issues/18373 should not be the casue. Would you help try with `scale=False` and see if there is anything wrong?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] wkcn edited a comment on issue #18475: BatchNorm can not converge with scale=False
Posted by GitBox <gi...@apache.org>.
wkcn edited a comment on issue #18475:
URL: https://github.com/apache/incubator-mxnet/issues/18475#issuecomment-639377361
I try to test the batch norm with `fix_gamma=True`.
The result on CPU is right, but that on GPU is wrong when `fix_gamma=True, axis = 1 and cudnn_off=False`.
Here are the failure cases when `fix_gamma=True`.
operator|shape|axis|cudnn_off|output_mean_var
--|--|--|--|--
BatchNorm|(24, 2)|1|False|False
BatchNorm|(24, 2)|1|False|True
BatchNorm|(24, 3, 4)|1|False|False
BatchNorm|(24, 3, 4)|1|False|True
BatchNorm|(24, 4, 4, 4)|1|False|False
BatchNorm|(24, 4, 4, 4)|1|False|True
BatchNorm|(24, 8, 4, 4)|1|False|False
BatchNorm|(24, 8, 4, 4)|1|False|True
BatchNorm|(24, 5, 6, 4, 4)|1|False|False
BatchNorm|(24, 5, 6, 4, 4)|1|False|True
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org