You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/03/08 21:04:36 UTC
[GitHub] Jerryzcn opened a new issue #10042: Gluon dataloader crash on speech recognition training
Jerryzcn opened a new issue #10042: Gluon dataloader crash on speech recognition training
URL: https://github.com/apache/incubator-mxnet/issues/10042
Note: Providing complete information in the most concise form is the best way to get help. This issue template serves as the checklist for essential information to most of the technical issues and bug reports. For non-technical issues and feature requests, feel free to present the information in what you believe is the best form.
For Q & A and discussion, please start a discussion thread at https://discuss.mxnet.io
## Description
(Brief description of the problem in no more than 2 sentences.)
Gluon data loader crash during training.
## Environment info (Required)
```
What to do:
1. Download the diagnosis script from https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py
2. Run the script using `python diagnose.py` and paste its output here.
----------Python Info----------
Version : 3.6.3
Compiler : GCC 7.2.0
Build : ('default', 'Oct 13 2017 12:02:49')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 9.0.1
Directory : /home/ubuntu/anaconda3/lib/python3.6/site-packages/pip
----------MXNet Info-----------
/home/ubuntu/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
/home/ubuntu/anaconda3/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py:46: DeprecationWarning: OpenSSL.rand is deprecated - you should use os.urandom instead
import OpenSSL.SSL
Version : 1.2.0
Directory : /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet
Hashtag not found. Not installed from pre-built package.
----------System Info----------
Platform : Linux-4.4.0-1052-aws-x86_64-with-debian-stretch-sid
system : Linux
node : ip-172-31-22-177
release : 4.4.0-1052-aws
version : #61-Ubuntu SMP Mon Feb 12 23:05:58 UTC 2018
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz
Stepping: 1
CPU MHz: 2699.984
CPU max MHz: 3000.0000
CPU min MHz: 1200.0000
BogoMIPS: 4600.07
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 46080K
NUMA node0 CPU(s): 0-15,32-47
NUMA node1 CPU(s): 16-31,48-63
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single retpoline kaiser fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx xsaveopt ida
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0022 sec, LOAD: 0.5053 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1427 sec, LOAD: 0.0642 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.1806 sec, LOAD: 0.1844 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0252 sec, LOAD: 0.1844 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0117 sec, LOAD: 0.2449 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0081 sec, LOAD: 0.3814 sec.
```
Package used (Python/R/Scala/Julia):
I'm using Python
For Scala user, please provide:
1. Java version: (`java -version`)
2. Maven version: (`mvn -version`)
3. Scala runtime if applicable: (`scala -version`)
For R user, please provide R `sessionInfo()`:
## Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
```
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.9' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
```
MXNet commit hash:
(Paste the output of `git rev-parse HEAD` here.)
2a9c7d9c7e1f2cdb49e029aef9b20e61afacbe46
Build config:
(Paste the content of config.mk, or the build command.)
make -j $(nproc) USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1
## Error Message:
(Paste the complete error message, including stack trace.)
```
Traceback (most recent call last):
File "/home/ubuntu/openspeech/openspeech/experiment.py", line 358, in <module>
exp.run()
File "/home/ubuntu/openspeech/openspeech/experiment.py", line 352, in run
stage_inst.run()
File "/home/ubuntu/openspeech/openspeech/train.py", line 140, in run
for step, data in enumerate(train_loader):
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/gluon/data/dataloader.py", line 224, in __iter__
idx, batch = data_queue.get()
File "/home/ubuntu/anaconda3/lib/python3.6/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/gluon/data/dataloader.py", line 38, in rebuild_ndarray
return nd.NDArray(nd.ndarray._new_from_shared_mem(*args))
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/ndarray/ndarray.py", line 151, in _new_from_shared_mem
ctypes.byref(hdl)))
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/base.py", line 149, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [20:48:23] src/storage/./cpu_shared_storage_manager.h:178: Check failed: ptr != ((void *) -1) (0xffffffffffffffff vs. 0xffffffffffffffff) Failed to map shared memory. mmap failed with error Cannot allocate memory
Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x5b) [0x7f77c7b896cb]
[bt] (1) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f77c7b8a208]
[bt] (2) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(mxnet::storage::CPUSharedStorageManager::Alloc(mxnet::Storage::Handle*)+0x99b) [0x7f77ca7ef5db]
[bt] (3) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(mxnet::StorageImpl::Alloc(mxnet::Storage::Handle*)+0x5d) [0x7f77ca7efd4d]
[bt] (4) /home/ubuntu/anaconda3/lib/python3.6/site-packages/mxnet-1.2.0-py3.6.egg/mxnet/libmxnet.so(MXNDArrayCreateFromSharedMem+0x825) [0x7f77ca848135]
[bt] (5) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call_unix64+0x4c) [0x7f7877dbaec0]
[bt] (6) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/../../libffi.so.6(ffi_call+0x22d) [0x7f7877dba87d]
[bt] (7) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2ce) [0x7f7877fcf82e]
[bt] (8) /home/ubuntu/anaconda3/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(+0x12265) [0x7f7877fd0265]
[bt] (9) python(_PyObject_FastCallDict+0x8b) [0x55a780a2554b]
```
## Minimum reproducible example
(If you are using your own code, please provide a short script that reproduces the error. Otherwise, please provide link to the existing example.)
## Steps to reproduce
(Paste the commands you ran that produced the error.)
It happens intermittently when we train on large speech dataset.
## What have you tried to solve it?
Not yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services