You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/09/22 21:58:46 UTC
[GitHub] [incubator-mxnet] johnbroughton2017 opened a new issue #16239:
train_imagenet.py fails with float16 + alexnet
johnbroughton2017 opened a new issue #16239: train_imagenet.py fails with float16 + alexnet
URL: https://github.com/apache/incubator-mxnet/issues/16239
## Description
I found the following error when training an alexnet using fp16. It is very easy reproduce using the command below. I have seen a few people reporting this bug yet I haven't been able to find a solution. If you replace alexnet with reset, it works fine.
## Error Message:
```
mxnet.base.MXNetError: [14:57:10] include/mxnet/././tensor_blob.h:236: Check failed: mshadow::DataType<DType>::kFlag == type_flag_: TBlob.get_with_shape: data type do not match specified type.Expected: 2 v.s. given 0
Stack trace:
[bt] (0) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x4a3b8b) [0x7f377f32ab8b]
[bt] (1) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x304366d) [0x7f3781eca66d]
[bt] (2) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x3553e98) [0x7f37823dae98]
[bt] (3) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2675018) [0x7f37814fc018]
[bt] (4) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x267abf5) [0x7f3781501bf5]
[bt] (5) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x265a8a1) [0x7f37814e18a1]
[bt] (6) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x265ddb0) [0x7f37814e4db0]
[bt] (7) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x265e046) [0x7f37814e5046]
[bt] (8) /home/xinghua/anaconda2/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2659004) [0x7f37814e0004]
```
## Minimum reproducible example
```
python train_imagenet.py --network alexnet --benchmark 1 --gpus 0 --batch-size 64 --dtype float16
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services