You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/02/04 16:20:27 UTC
[GitHub] [incubator-mxnet] matteosal opened a new issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
matteosal opened a new issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841
## Description
Some specific symbol evaluation fail with a CUDA error. I have reproduced this problem on:
* mxnet 1.6 + CUDA 10.1
* mxnet 1.6 + CUDA 11.2
* mxnet 1.7 + CUDA 11.2
(didn't try mxnet 1.7 + CUDA 10.1). My GPU is a GTX 1650 Mobile.
### Error Message
```
mxnet.base.MXNetError: Traceback (most recent call last):
[bt] (11) /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7fcfe137d293]
[bt] (10) /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7fcfe1241609]
[bt] (9) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6d84) [0x7fcfcfe04d84]
[bt] (8) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void (std::shared_ptr<dmlc::ManualEvent>)>, std::shared_ptr<dmlc::ManualEvent> > > >::_M_run()+0x4a) [0x7fcf900d826a]
[bt] (7) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(std::_Function_handler<void (std::shared_ptr<dmlc::ManualEvent>), mxnet::engine::ThreadedEnginePerDevice::PushToExecute(mxnet::engine::OprBlock*, bool)::{lambda()#4}::operator()() const::{lambda(std::shared_ptr<dmlc::ManualEvent>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<dmlc::ManualEvent>&&)+0x4e) [0x7fcf900dcb9e]
[bt] (6) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(void mxnet::engine::ThreadedEnginePerDevice::GPUWorker<(dmlc::ConcurrentQueueType)0>(mxnet::Context, bool, mxnet::engine::ThreadedEnginePerDevice::ThreadWorkerBlock<(dmlc::ConcurrentQueueType)0>*, std::shared_ptr<dmlc::ManualEvent> const&)+0x11d) [0x7fcf900dc8fd]
[bt] (5) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::engine::ThreadedEngine::ExecuteOprBlock(mxnet::RunContext, mxnet::engine::OprBlock*)+0x11f) [0x7fcf900d91ff]
[bt] (4) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(+0x18ae22a) [0x7fcf900ff22a]
[bt] (3) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::exec::FComputeExecutor::Run(mxnet::RunContext, bool)+0x6d) [0x7fcf900f143d]
[bt] (2) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(void mxnet::FusedOp::Forward<mshadow::gpu>(nnvm::NodeAttrs const&, mxnet::OpContext const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&, std::vector<mxnet::OpReqType, std::allocator<mxnet::OpReqType> > const&, std::vector<mxnet::TBlob, std::allocator<mxnet::TBlob> > const&)+0x9e5) [0x7fcf938e6585]
[bt] (1) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(mxnet::FusedOp::CompileCode(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x1228) [0x7fcf938de8c8]
[bt] (0) /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x72) [0x7fcf8ff79e92]
File "/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/src/operator/fusion/fused_op.cu", line 672
MXNetError: CUDA Driver: an illegal memory access was encountered
```
## To Reproduce
Download this json file then run the script:
[sym.zip](https://github.com/apache/incubator-mxnet/files/5926995/sym.zip)
```
import mxnet as mx
ctx = mx.gpu()
sym = mx.symbol.load('path/to/sym.json')
ex = sym.bind(ctx, args={"input": mx.nd.array([[1, 2, 3, 4]], ctx=ctx)})
outputs = ex.forward()
print(outputs)
```
## What have you tried to solve it?
I'm building mxnet from source with cmake, and I also happen to be in possession of a 1.6 + CUDA 10.1 build (which is one of the combinations I've tried) which **doesn't** have this problem (but I don't have all the details on the settings/environment which produced it). This suggests that the problem can be fixed by some build settings, however I have spent significant time rebuilding with various settings and never managed to get anything. These are my cmake settings:
```
cmake \
`# GENERAL FLAGS` \
-DCMAKE_INSTALL_PREFIX=$output_dir \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_SKIP_BUILD_RPATH=On \
-DUSE_OPENCV=OFF \
-DUSE_F16C=Off `# float16 support`\
-DUSE_INT64_TENSOR_SIZE=On \
-DCMAKE_C_FLAGS_RELEASE="-DNDEBUG" \
-DCMAKE_CXX_FLAGS_RELEASE="-DNDEBUG --std=c++14" \
`# MATH BACKENDS` \
-DBLAS=MKL \
-DUSE_LAPACK=OFF \
-DUSE_MKLDNN=OFF \
-DUSE_MKL_IF_AVAILABLE=ON \
-DMKL_USE_ILP64=ON \
-DMKL_USE_SINGLE_DYNAMIC_LIBRARY=OFF \
-DMKL_ROOT=$mkl_dir \
-DMKL_INCLUDE_DIR=$mkl_dir \
-DMKL_RT_LIBRARY="$mkl_dir/libmkl_def.so;$mkl_dir/libmkl_intel_ilp64.so;$mkl_dir/libmkl_core.so;$mkl_dir/libmkl_intel_thread.so;$mkl_dir/libiomp5.so" \
`# OPENMP` \
-DUSE_OPENMP=ON \
-DOpenMP_C_FLAGS="-I$mkl_dir" \
-DOpenMP_C_LIB_NAMES="libiomp5" \
-DOpenMP_CXX_FLAGS="-I$mkl_dir" \
-DOpenMP_CXX_LIB_NAMES="libiomp5" \
-DOpenMP_libiomp5_LIBRARY="$mkl_dir/libiomp5.so" \
`# CUDA` \
-DUSE_CUDA=ON \
-DUSE_CUDNN=ON \
-DCUDNN_LIBRARY=$cudnn_dir/lib64/libcudnn.so.8 \
-DCUDNN_INCLUDE=$cudnn_dir/include \
-DUSE_NCCL=OFF \
-DCUDNN_ROOT:PATH=$cudnn_dir \
-DMXNET_CUDA_ARCH="7.5+PTX" \
-DCUDAToolkit_ROOT=$cuda_dir \
-DCMAKE_CUDA_COMPILER:PATH=$cuda_dir/bin/nvcc \
```
## Environment
<details>
<summary>Environment Information</summary>
```
----------Python Info----------
Version : 3.8.5
Compiler : GCC 9.3.0
Build : ('default', 'Jul 28 2020 12:59:40')
Arch : ('64bit', 'ELF')
------------Pip Info-----------
Version : 20.0.2
Directory : /usr/lib/python3/dist-packages/pip
----------MXNet Info-----------
Version : 1.7.0
Directory : /home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet
Commit hash file "/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/COMMIT_HASH" not found. Not installed from pre-built package or built from source.
Library : ['/home/matteo/Git/mxnet-build/Build/Linux-x86-64/CUDA/mxnet/python/mxnet/../../lib/libmxnet.so']
Build features:
✔ CUDA
✔ CUDNN
✖ NCCL
✔ CUDA_RTC
✖ TENSORRT
✔ CPU_SSE
✔ CPU_SSE2
✔ CPU_SSE3
✖ CPU_SSE4_1
✖ CPU_SSE4_2
✖ CPU_SSE4A
✖ CPU_AVX
✖ CPU_AVX2
✔ OPENMP
✖ SSE
✖ F16C
✖ JEMALLOC
✖ BLAS_OPEN
✖ BLAS_ATLAS
✔ BLAS_MKL
✖ BLAS_APPLE
✖ LAPACK
✖ MKLDNN
✖ OPENCV
✖ CAFFE
✖ PROFILER
✖ DIST_KVSTORE
✖ CXX14
✔ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
✖ TVM_OP
----------System Info----------
Platform : Linux-5.8.0-41-generic-x86_64-with-glibc2.29
system : Linux
node : pajarulo
release : 5.8.0-41-generic
version : #46~20.04.1-Ubuntu SMP Mon Jan 18 17:52:23 UTC 2021
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 158
Model name: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Stepping: 13
CPU MHz: 4161.311
CPU max MHz: 5000,0000
CPU min MHz: 800,0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 256 KiB
L1i cache: 256 KiB
L2 cache: 2 MiB
L3 cache: 16 MiB
NUMA node0 CPU(s): 0-15
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds: Mitigation; TSX disabled
Vulnerability Tsx async abort: Mitigation; TSX disabled
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch
_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2
apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi
flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat
pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-773434293
Can you try with the latest `v1.x` version? For example, https://repo.mxnet.io/dist/python/cu110/mxnet_cu110-2.0.0b20210202-py3-none-manylinux2014_x86_64.whl
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] TristonC commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
TristonC commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-774382923
Seems the `nvrtcGetCUBIN' , `nvrtcGetCUBINSize' were introduced in CUDA 11.1+. Did you install CUDA10.2 or CUDA11.2 in your system?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-773434293
Can you try with the latest `v1.x` version? For example, https://repo.mxnet.io/dist/python/cu110/mxnet_cu110-2.0.0b20210202-py3-none-manylinux2014_x86_64.whl
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-774252698
For your build settings: Do you need ILP64? Otherwise I suggest you disable DUSE_INT64_TENSOR_SIZE and ILP64.
Further, you don't need +PTX for the cuda arch. Just provide the correct Arch number for your gpu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] TristonC commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
TristonC commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-773480987
Could you try v1.x with CUDA 11.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] matteosal edited a comment on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
matteosal edited a comment on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-774244798
* The version at the link provided by @leezu works (however `mxnet.__version__` is 2.0.0 on that package and not 1.x)
* I have tried to build the latest `v1.x` branch from source and I still find the same error
* I have tried to build v2.0 (master) but the build failed to link with
```
/usr/bin/ld: ../libmxnet.so: undefined reference to `nvrtcGetCUBINSize'
/usr/bin/ld: ../libmxnet.so: undefined reference to `nvrtcGetCUBIN'
```
Anyway, let me repeat that I don't think this is a problem with the source code, but with the build procedure (and building from source is a hard requirement for what I'm doing). As I have stated in the report, I am in possession of a 1.6 + CUDA 10.1 build (which is one of the combinations I've tried) which doesn't have this problem, but unfortunately I don't know the exact details/settings of that build. My build settings are in the report, what could be wrong there?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] matteosal commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
matteosal commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-774244798
* The version at the link provided by @leezu works (however `mxnet.__version__` is 2.0.0 on that package and not 1.x)
* I have tried to build the latest `v1.x` branch from source and I still find the same error
* I have tried to build v2.0 (master) but the build failed to link with
```
/usr/bin/ld: ../libmxnet.so: undefined reference to `nvrtcGetCUBINSize'
/usr/bin/ld: ../libmxnet.so: undefined reference to `nvrtcGetCUBIN'
```
Anyway, let me repeat that I don't think this is a problem with the source code, but with the build procedure. As I have stated in the report, I am in possession of a 1.6 + CUDA 10.1 build (which is one of the combinations I've tried) which doesn't have this problem, but unfortunately I don't know the exact details/settings of that build. My build settings are in the report, what could be wrong there?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] matteosal commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
matteosal commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-775094886
@leezu switching off `USE_INT64_TENSOR_SIZE` fixes the problem!
@TristonC I had both installed to make my tests and it's possible that I have set some path wrong when building/running v 2.0. Anyway, this doesn't matter anymore.
Anyway, this looks like a bug with `USE_INT64_TENSOR_SIZE` right? Should I open a separate issue for this?
Thank you
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] TristonC commented on issue #19841: GPU computation failing with CUDA Driver: an illegal memory access was encountered (Linux)
Posted by GitBox <gi...@apache.org>.
TristonC commented on issue #19841:
URL: https://github.com/apache/incubator-mxnet/issues/19841#issuecomment-773480987
Could you try v1.x with CUDA 11.1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org