[GitHub] kohr-h opened a new issue #13135: [Python] CUDNN error from 3D deconvolution

[GitBox <gi...@apache.org>]

kohr-h opened a new issue #13135: [Python] CUDNN error from 3D deconvolution
URL: https://github.com/apache/incubator-mxnet/issues/13135
 
 
   ## Description
   
   When I build a simple net (Gluon interface) that involves a 3D deconvolution, and run it on some sample data, I get an error indicating that CUDNN couldn't find an algorithm.
   
   ## Environment info (Required)
   
   ```
   ----------Python Info----------
   Version      : 3.6.7
   Compiler     : MSC v.1915 64 bit (AMD64)
   Build        : ('default', 'Oct 28 2018 19:44:12')
   Arch         : ('64bit', 'WindowsPE')
   ------------Pip Info-----------
   Version      : 18.1
   Directory    : C:\Users\Holger\AppData\Local\conda\conda\envs\mxnet\lib\site-packages\pip
   ----------MXNet Info-----------
   Version      : 1.3.1
   Directory    : c:\users\holger\git\mxnet\python\mxnet
   Hashtag not found. Not installed from pre-built package.
   ----------System Info----------
   Platform     : Windows-10-10.0.17134-SP0
   system       : Windows
   node         : DESKTOP-3DBNGT7
   release      : 10
   version      : 10.0.17134
   ----------Hardware Info----------
   machine      : AMD64
   processor    : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel
   Name
   Intel(R) Xeon(R) W-2175 CPU @ 2.50GHz
   ```
   
   Package used (Python/R/Scala/Julia): Python
   
   ## Build info (Required if built from source)
   
   Compiler (gcc/clang/mingw/visual studio): VC++ 14.11
   
   MXNet commit hash: a91b364c490b163da424a1c2935652ea7b799050
   
   Build config: 
   
   - CUDA 9.2
   - CUDNN 7.3
   - no MKL-DNN
   
   ## Error Message:
   
   ```
   [14:23:43] c:\users\holger\git\mxnet\src\operator\nn\cudnn\./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
   [14:23:43] c:\users\holger\git\mxnet\src\operator\nn\cudnn\./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
   Traceback (most recent call last):
     File "tmp.py", line 15, in <module>
       output_npy = output.asnumpy()
     File "c:\users\holger\git\mxnet\python\mxnet\ndarray\ndarray.py", line 1980, in asnumpy
       ctypes.c_size_t(data.size)))
     File "c:\users\holger\git\mxnet\python\mxnet\base.py", line 252, in check_call
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   mxnet.base.MXNetError: [14:23:43] c:\users\holger\git\mxnet\src\operator\nn\./cudnn/cudnn_deconvolution-inl.h:849: Failed to find any forward deconvolution algorithm with workspace size of 536870912 bytes, please consider reducing batch/model size or increasing the workspace size
   ```
   
   ## Minimum reproducible example
   
   ```py
   import mxnet as mx
   from mxnet import nd
   from mxnet.gluon import nn
   
   net = nn.Sequential()
   net.add(nn.Conv3D(1, (3, 3, 3), padding=(1, 1, 1)))
   # Without the following line, everything is okay
   net.add(nn.Conv3DTranspose(1, (3, 3, 3), padding=(1, 1, 1)))
   
   net.initialize(ctx=mx.gpu(0))
   
   input = nd.zeros((1, 1, 64, 64, 64), ctx=mx.gpu(0))
   output = net(input)
   
   input_npy = input.asnumpy()
   output_npy = output.asnumpy()
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services