You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/06/12 06:32:41 UTC
[GitHub] eric-haibin-lin opened a new issue #11241: Conv1D throws
CUDNN_STATUS_EXECUTION_FAILED
eric-haibin-lin opened a new issue #11241: Conv1D throws CUDNN_STATUS_EXECUTION_FAILED
URL: https://github.com/apache/incubator-mxnet/issues/11241
Setup:
```
$ pip install mxnet-cu90==1.1.0
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
$ ls /usr/local/cuda/lib64/libcudnn.so.7.0.3
/usr/local/cuda/lib64/libcudnn.so.7.0.3
```
Run the following script:
```
W_REQ = 'add'
shape = (1, 65536, 1)
ctx = mx.gpu()
kwargs = {'no_bias': True, 'kernel': (1,), 'num_filter': 1}
x = mx.sym.var('x')
w = mx.sym.var('w')
x_grad = mx.nd.zeros(shape, ctx=ctx)
w_grad = mx.nd.zeros(shape, ctx=ctx)
args_grad = {'x': x_grad, 'w': w_grad}
sym = mx.sym.Convolution(x, w, **kwargs)
executor = sym.bind(ctx, grad_req={'x': 'null', 'w': W_REQ},
args={'x': mx.nd.ones(shape, ctx=ctx), 'w': mx.nd.ones(shape, ctx=ctx)},
args_grad=args_grad)
executor.forward()
executor.backward([mx.nd.ones((1,1,1), ctx=ctx)])
mx.nd.waitall()
```
Gives the following error:
```
[06:31:41] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
terminate called after throwing an instance of 'dmlc::Error'
what(): [06:31:41] src/engine/./threaded_engine.h:359: [06:31:41] src/operator/nn/./cudnn/cudnn_convolution-inl.h:242: Check failed: e == CUDNN_STATUS_SUCCESS (8 vs. 0) cuDNN: CUDNN_STATUS_EXECUTION_FAILED
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a920d1) [0x7f706018c0d1]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x262f5e7) [0x7f705fd295e7]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x24570bb) [0x7f705fb510bb]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x245d7d4) [0x7f705fb577d4]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e2ed) [0x7f705fb382ed]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6]
[bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b]
A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 9 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e594) [0x7f705fb38594]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f70d400bc80]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f70d52a66ba]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f70d4fdc3dd]
```
Note that there's no error if `W_REQ` is changed to 'write'.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services