You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/08/10 18:33:13 UTC
[GitHub] prathik-naidu opened a new issue #12127: LSTM split0 Operator Error
prathik-naidu opened a new issue #12127: LSTM split0 Operator Error
URL: https://github.com/apache/incubator-mxnet/issues/12127
## Description
Getting an error in the split0 operator when training an image captioning network in mxnet.
## Environment info (Required)
```
----------Python Info----------
Version : 3.6.4
Compiler : GCC 7.2.0
Build : ('default', 'Mar 13 2018 01:15:57')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 9.0.1
Directory : /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version : 1.1.0
Directory : /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet
Commit Hash : 07a83a0325a3d782513a04f47d711710972cb144
----------System Info----------
Platform : Linux-4.4.0-1062-aws-x86_64-with-debian-stretch-sid
system : Linux
node : ip-172-31-5-176
release : 4.4.0-1062-aws
version : #71-Ubuntu SMP Fri Jun 15 10:07:39 UTC 2018
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping: 1
CPU MHz: 2699.804
CPU max MHz: 3000.0000
CPU min MHz: 1200.0000
BogoMIPS: 4600.09
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 46080K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0060 sec, LOAD: 0.4613 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0701 sec, LOAD: 0.4250 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1166 sec, LOAD: 0.3497 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0044 sec, LOAD: 0.1650 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0039 sec, LOAD: 0.0744 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0035 sec, LOAD: 0.1032 sec.
```
Package used (Python/R/Scala/Julia): I'm using Python
## Error Message:
```
---------------INFO-----------------------
vocab_size:663
sentence_length:46
-----------------------------------------
Creating Iterators...
Initiating Training...
INFO:root:Epoch[0] Train-perplexity=655.513238
INFO:root:Epoch[0] Time cost=1.261
infer_shape error. Arguments:
image_feature: (50, 1024)
word_data: (50, 77)
softmax_label: (50,)
Traceback (most recent call last):
File "2_train_val.py", line 102, in <module>
epoch_end_callback=mx.callback.do_checkpoint(checkpoints_prefix, period=10)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/module/base_module.py", line 528, in fit
batch_end_callback=eval_batch_end_callback, epoch=epoch)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/module/base_module.py", line 244, in score
self.forward(eval_batch, is_train=False)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/module/module.py", line 608, in forward
self.reshape(new_dshape, new_lshape)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/module/module.py", line 470, in reshape
self._exec_group.reshape(self._data_shapes, self._label_shapes)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/module/executor_group.py", line 381, in reshape
self.bind_exec(data_shapes, label_shapes, reshape=True)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/module/executor_group.py", line 357, in bind_exec
allow_up_sizing=True, **dict(data_shapes_i + label_shapes_i))
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/executor.py", line 402, in reshape
arg_shapes, _, aux_shapes = self._symbol.infer_shape(**kwargs)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 989, in infer_shape
res = self._infer_shape_impl(False, *args, **kwargs)
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/symbol/symbol.py", line 1119, in _infer_shape_impl
ctypes.byref(complete)))
File "/home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/base.py", line 146, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator split0: [18:11:40] src/operator/./slice_channel-inl.h:208: Check failed: dshape[real_axis] % param_.num_outputs == 0U (31 vs. 0) You are trying to split the 1-th axis of input tensor with shape [50,78,256] into num_outputs=47 evenly sized chunks, but this is not possible because 47 does not evenly divide 78
Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x276938) [0x7f446310f938]
[bt] (1) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x276d48) [0x7f446310fd48]
[bt] (2) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x286ccb7) [0x7f4465705cb7]
[bt] (3) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x25eab07) [0x7f4465483b07]
[bt] (4) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x244274f) [0x7f44652db74f]
[bt] (5) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(+0x2445268) [0x7f44652de268]
[bt] (6) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/libmxnet.so(MXSymbolInferShape+0x1539) [0x7f4465260659]
[bt] (7) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f4487cd1ec0]
[bt] (8) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f4487cd187d]
[bt] (9) /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f4487ee6dee]
```
## Minimum reproducible example
I'm using the code from the following repository: [https://github.com/saicoco/mxnet_image_caption](https://github.com/saicoco/mxnet_image_caption)
Everything is identical, except I use my own dataset. I've preprocessed the data identical to what this implementation expects (originally used the Flickr8k dataset).
## What have you tried to solve it?
I basically need some help trying to understand where this error is coming from – in particular, why `param_.num_output` is set to 47.
1. The error message is thrown here: [https://github.com/apache/incubator-mxnet/blob/master/src/operator/slice_channel-inl.h#L208](https://github.com/apache/incubator-mxnet/blob/master/src/operator/slice_channel-inl.h#L208)
2. `num_outputs` seems to be set here: [https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/module/base_module.py#L385](https://github.com/apache/incubator-mxnet/blob/master/python/mxnet/module/base_module.py#L385) ... although this happens after self.forward is called, but the error message seems to be thrown before `num_outputs` is set
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services