[GitHub] hallazie opened a new issue #12493: ctc_loss with large alphabet size raises CUDA error

[GitBox <gi...@apache.org>]

hallazie opened a new issue #12493: ctc_loss with large alphabet size raises CUDA error
URL: https://github.com/apache/incubator-mxnet/issues/12493
 
 
   ##Enviroment:
   python2.7/windows7_64bit
   mxnet-1.2.0
   Nvidia Driver Version 397.31
   
   ## Error Message:
   [15:41:28] G:\deeplearn\mxnet\dmlc-core\include\dmlc/logging.h:308: [15:41:28] g:\deeplearn\mxnet\mshadow\mshadow\./stream_gpu-inl.h:62: Check failed: e == cudaSuccess CUDA: unknown error
   [15:41:28] G:\deeplearn\mxnet\dmlc-core\include\dmlc/logging.h:308: [15:41:28] g:\deeplearn\mxnet\src\engine\./threaded_engine.h:370: [15:41:28] g:\deeplearn\mxnet\mshadow\mshadow\./stream_gpu-inl.h:62: Check failed: e == cudaSuccess CUDA: unknown error
   
   ## Minimum reproducible example
   ```python
   import mxnet as mx
   import numpy as np
   ctx=mx.gpu(0)
   alphabet_size=3000
   in_var = mx.sym.Variable('data')
   labels_var = mx.sym.Variable('label')
   ctc = mx.sym.contrib.ctc_loss(in_var, labels_var)
   loss = mx.symbol.MakeLoss(ctc)
   arg_shapes,_,_ = loss.infer_shape(data=(6,2,alphabet_size), label=(2,3))
   arg_array = [mx.nd.normal(shape=shape, ctx=ctx) for shape in arg_shapes]
   exe = loss.bind(ctx=ctx, args=arg_array)
   exe.forward(is_train=True)
   exe.backward()
   outTest = exe.outputs[0]
   print '%s'%(outTest.asnumpy())
   ```
   
   when `alphabet_size=200` the code works fine, when `alphabet_size=3000` (for chinese ocr task) the code crashes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services