[GitHub] marcoabreu opened a new issue #11395: Check failed: e == cudaSuccess CUDA: unspecified launch failure

[GitBox <gi...@apache.org>]

marcoabreu opened a new issue #11395: Check failed: e == cudaSuccess CUDA: unspecified launch failure
URL: https://github.com/apache/incubator-mxnet/issues/11395
 
 
   Sometimes, our slaves get corrupted and suddenly all test start to fail. This is unrelated to the tests directly.
   
   http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/incubator-mxnet/detail/PR-11377/5/pipeline/
   
   ```
   ======================================================================
   
   ERROR: test_operator_gpu.test_op_roi_align
   
   ----------------------------------------------------------------------
   
   Traceback (most recent call last):
   
     File "C:\Anaconda3\envs\py2\lib\site-packages\nose\case.py", line 197, in runTest
   
       self.test(*self.arg)
   
     File "C:\Anaconda3\envs\py2\lib\site-packages\nose\util.py", line 620, in newfunc
   
       return func(*arg, **kw)
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\common.py", line 157, in test_new
   
       orig_test(*args, **kwargs)
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\test_operator.py", line 6269, in test_op_roi_align
   
       test_roi_align_value()
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\tests\python\gpu\../unittest\test_operator.py", line 6230, in test_roi_align_value
   
       data = mx.nd.array(np.arange(N*C*W*H).reshape((N,C,H,W)), ctx=ctx, dtype = dtype)
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\pkg_vc14_gpu\python\mxnet\ndarray\utils.py", line 146, in array
   
       return _array(source_array, ctx=ctx, dtype=dtype)
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\pkg_vc14_gpu\python\mxnet\ndarray\ndarray.py", line 2357, in array
   
       arr[:] = source_array
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\pkg_vc14_gpu\python\mxnet\ndarray\ndarray.py", line 444, in __setitem__
   
       self._set_nd_basic_indexing(key, value)
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\pkg_vc14_gpu\python\mxnet\ndarray\ndarray.py", line 710, in _set_nd_basic_indexing
   
       self._sync_copyfrom(value)
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\pkg_vc14_gpu\python\mxnet\ndarray\ndarray.py", line 876, in _sync_copyfrom
   
       ctypes.c_size_t(source_array.size)))
   
     File "C:\jenkins_slave\workspace\ut-python-gpu\pkg_vc14_gpu\python\mxnet\base.py", line 210, in check_call
   
       raise MXNetError(py_str(_LIB.MXGetLastError()))
   
   MXNetError: [06:35:08] c:\jenkins_slave\workspace\build-gpu\3rdparty\mshadow\mshadow\./tensor_gpu-inl.h:69: Check failed: e == cudaSuccess CUDA: unspecified launch failure
   
   -------------------- >> begin captured logging << --------------------
   
   common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1046236735 to reproduce.
   
   --------------------- >> end captured logging << ---------------------
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services