You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/10/08 21:48:05 UTC
[GitHub] [incubator-mxnet] leezu opened a new issue #16399: ndarray.load
crashes MXNet GPU builds on CPU machines
leezu opened a new issue #16399: ndarray.load crashes MXNet GPU builds on CPU machines
URL: https://github.com/apache/incubator-mxnet/issues/16399
## Description
For some parameter files below crash happens. It does not happen for all parameter files.
## Environment info (Required)
```
----------Python Info----------
Version : 3.7.3
Compiler : GCC 5.4.0 20160609
Build : ('default', 'Jun 13 2019 13:24:27')
Arch : ('64bit', 'ELF')
------------Pip Info-----------
Version : 19.0.3
Directory : /home/ubuntu/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip
----------MXNet Info-----------
Version : 1.5.1
Directory : /home/ubuntu/.local/lib/python3.7/site-packages/mxnet
Commit Hash : c9818480680f84daa6e281a974ab263691302ba8
Library : ['/home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so']
Build features:
✔ CUDA
✔ CUDNN
✔ NCCL
✔ CUDA_RTC
✖ TENSORRT
✔ CPU_SSE
✔ CPU_SSE2
✔ CPU_SSE3
✔ CPU_SSE4_1
✔ CPU_SSE4_2
✖ CPU_SSE4A
✔ CPU_AVX
✖ CPU_AVX2
✖ OPENMP
✖ SSE
✔ F16C
✖ JEMALLOC
✖ BLAS_OPEN
✖ BLAS_ATLAS
✖ BLAS_MKL
✖ BLAS_APPLE
✔ LAPACK
✖ MKLDNN
✔ OPENCV
✖ CAFFE
✖ PROFILER
✔ DIST_KVSTORE
✖ CXX14
✖ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
----------System Info----------
Platform : Linux-4.4.0-1095-aws-x86_64-with-debian-stretch-sid
system : Linux
node : ip-172-31-84-3
release : 4.4.0-1095-aws
version : #106-Ubuntu SMP Wed Sep 18 13:33:48 UTC 2019
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
Stepping: 4
CPU MHz: 2499.998
BogoMIPS: 4999.99
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 33792K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single kaiser fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 ida arat pku
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0010 sec, LOAD: 0.3120 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.3964 sec, LOAD: 0.0625 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0938 sec, LOAD: 0.3685 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0266 sec, LOAD: 0.1123 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0007 sec, LOAD: 0.0284 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0054 sec, LOAD: 0.0288 sec.
----------Environment----------
KMP_DUPLICATE_LIB_OK="True"
```
Package used (Python/R/Scala/Julia):
(I'm using ...)
## Error Message:
```
what(): [21:44:06] src/engine/threaded_engine.cc:328: Check failed: device_count_ > 0 (-1 vs. 0) : GPU usage requires at least 1 GPU
Stack trace:
[bt] (0) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x4b04cb) [0x7fab268524cb]
[bt] (1) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25b50ec) [0x7fab289570ec]
[bt] (2) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x25b5cd4) [0x7fab28957cd4]
[bt] (3) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::NDArray::Chunk::~Chunk()+0x3c2) [0x7fab28b57022]
[bt] (4) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::NDArray::~NDArray()+0x9a) [0x7fab2698422a]
[bt] (5) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::NDArray::Copy(mxnet::Context) const+0x2cd) [0x7fab28b7651d]
[bt] (6) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::NDArray::Load(dmlc::Stream*)+0xb41) [0x7fab28b78211]
[bt] (7) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::NDArray::Load(dmlc::Stream*, std::vector<mxnet::NDArray, std::allocator<mxnet::NDArray> >*, std::vector<std::string, std::allocator<std::string> >*)+0xc51) [0x7fab28b78eb1]
[bt] (8) /home/ubuntu/.local/lib/python3.7/site-packages/mxnet/libmxnet.so(MXNDArrayLoad+0x263) [0x7fab288cf613]
```
## Minimum reproducible example
``` bash
wget http://apache-mxnet.s3.amazonaws.com/gluon/models/awd_lstm_lm_1150_wikitext-2-f9562ed0.zip
unzip awd_lstm_lm_1150_wikitext-2-f9562ed0.zip
python3 -c 'import mxnet as mx; mx.nd.load("awd_lstm_lm_1150_wikitext-2-f9562ed0.params")'
```
## Steps to reproduce
- `pip install mxnet-cu100` on a CPU machine with cuda installed
- Run above example
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services