You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/10/02 21:27:11 UTC
[GitHub] [incubator-mxnet] roywei opened a new issue #11774: Gluon BatchNorm
beta=False not working properly
roywei opened a new issue #11774: Gluon BatchNorm beta=False not working properly
URL: https://github.com/apache/incubator-mxnet/issues/11774
**Description**
I came from #10401
In that issue, there is no reproducible code, so not sure whats the use case there.
When I was developing mxnet backend for keras, I have similar issue.
So I used this [unit test](https://github.com/keras-team/keras/blob/6489c457d03e5fd5d605ed6197932f3b769ff951/tests/keras/layers/normalization_test.py#L144) and was able to reproduce:
**Config**
machine: mac
mxnet: latest master (pip install mxnet --pre)
keras: 2.2.0 with TensorFlow backend
**The purpose of the test:**
Create a random normalized data with size size=(1000, 3, 4, 4), centered on 5.0, variance 10.0.
Use batchnorm layer to normalize it to have mean close to 0, and std close to 1.
**Reproducible code:**
First part of the code shows Keras implementation with TensorFlow backend, you can see mean is close to 0 and std is close to 1.
Second Part of the code shows Gluon implementation, When scale and center is False, it throws error
```
import mxnet as mx
import numpy as np
from keras.layers import normalization
from keras.models import Sequential
from mxnet import autograd
from mxnet import gluon
from mxnet import nd
from numpy.testing import assert_allclose
"""
Keras test
"""
model = Sequential()
norm = normalization.BatchNormalization(center=False, scale=False,
input_shape=(3, 4, 4))
model.add(norm)
model.compile(loss='mse', optimizer='sgd')
# centered on 5.0, variance 10.0
x = np.random.normal(loc=5.0, scale=10.0, size=(1000, 3, 4, 4))
model.fit(x, x, epochs=4, verbose=0, batch_size=32)
out = model.predict(x)
print("Results from keras:")
print( "Mean: %s" % np.mean(out, axis=(0, 2, 3)))
print( "Std: %s" % np.std(out, axis=(0, 2, 3)))
assert_allclose(np.mean(out, axis=(0, 2, 3)), 0.0, atol=1e-1)
assert_allclose(np.std(out, axis=(0, 2, 3)), 1.0, atol=1e-1)
"""
Gluon test
"""
print(mx.__version__)
x = np.random.normal(loc=5.0, scale=10.0, size=(1000, 3, 4, 4))
x_nd = nd.array(x)
net = gluon.nn.Sequential()
ctx = mx.cpu()
with net.name_scope():
net.add(gluon.nn.BatchNorm(scale=False, center=False))
net.collect_params().initialize(mx.init.Normal(sigma=.1), ctx=ctx)
mse = gluon.loss.L2Loss()
trainer = gluon.Trainer(net.collect_params(), 'sgd')
epochs = 4
for e in range(epochs):
cumulative_loss = 0
for i in range(1000):
data = x_nd[i, :]
label = x_nd[i, :]
with autograd.record():
output = net(data)
loss = mse(output, label)
loss.backward(retain_graph=True)
trainer.step(32)
cumulative_loss += nd.sum(loss).asscalar()
out = net(x_nd).asnumpy()
print("Results from gluon:")
print( "Mean: %s" % np.mean(out, axis=(0, 2, 3)))
print( "Std: %s" % np.std(out, axis=(0, 2, 3)))
assert_allclose(np.mean(out, axis=(0, 2, 3)), 0.0, atol=1e-1)
assert_allclose(np.std(out, axis=(0, 2, 3)), 1.0, atol=1e-1)
```
**Error Message:**
```
Traceback (most recent call last):
File "/Users/lawei/Documents/Notebooks/keras/gluon_batchnorm.py", line 56, in <module>
loss.backward(retain_graph=True)
File "/Users/lawei/anaconda3/lib/python3.6/site-packages/mxnet/ndarray/ndarray.py", line 2130, in backward
ctypes.c_void_p(0)))
File "/Users/lawei/anaconda3/lib/python3.6/site-packages/mxnet/base.py", line 210, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [16:31:24] src/imperative/imperative.cc:285: Check failed: !AGInfo::IsNone(*i) Cannot differentiate node because it is not in a computational graph. You need to set is_recording to true or use autograd.record() to save computational graphs for backward. If you want to differentiate the same graph twice, you need to pass retain_graph=True to backward.
Stack trace returned 6 entries:
[bt] (0) 0 libmxnet.so 0x0000000111dfdeb4 libmxnet.so + 20148
[bt] (1) 1 libmxnet.so 0x0000000111dfdc6f libmxnet.so + 19567
[bt] (2) 2 libmxnet.so 0x0000000112f69389 MXNDListFree + 548393
[bt] (3) 3 libmxnet.so 0x0000000112ebe1fd MXAutogradBackwardEx + 893
[bt] (4) 4 libffi.6.dylib 0x0000000110670884 ffi_call_unix64 + 76
[bt] (5) 5 ??? 0x00007fff4fcabf40 0x0 + 140734532075328
```
**My Questions:**
1. gluon.nn.BatchNorm is not working with scale and beta both to False, need to fix
2. When changing scale=False, center=True, I was able to get std close to 1, but mean is not close to 0, see following output. What's the workaround to normalize this data to mean is 0, and std is 1. (I assume the current BatchNorm implementation is correct, some logic not handled when beta is None/False)
3. How to do this in Symbolic API?
```
Results from keras:
Mean: [-0.00930524 0.00754305 0.0025016 ]
Std: [ 0.99344313 0.99861413 1.00193286]
1.3.0
Results from gluon:
Mean: [ 3.16370535 2.95543957 3.03811169]
Std: [ 1.04609287 0.93967396 0.98776978]
Traceback (most recent call last):
File "/Users/lawei/Documents/Notebooks/keras/gluon_batchnorm.py", line 65, in <module>
assert_allclose(np.mean(out, axis=(0, 2, 3)), 0.0, atol=1e-1)
File "/Users/lawei/anaconda3/lib/python3.6/site-packages/numpy/testing/utils.py", line 1395, in assert_allclose
verbose=verbose, header=header, equal_nan=equal_nan)
File "/Users/lawei/anaconda3/lib/python3.6/site-packages/numpy/testing/utils.py", line 778, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0.1
(mismatch 100.0%)
x: array([ 3.163705, 2.95544 , 3.038112], dtype=float32)
y: array(0.0)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services