[GitHub] [tvm] masahi edited a comment on pull request #7123: Parallelize cumsum in get_valid_counts

[GitBox <gi...@apache.org>]

masahi edited a comment on pull request #7123:
URL: https://github.com/apache/tvm/pull/7123#issuecomment-754891319


   I can reproduce the issue by running ssd test in tensorflow/test_forward.py with cuda target (I looked at this test yesterday for my PR, so I have a fresh memory):
   
   ```
   terminate called after throwing an instance of 'dmlc::Error'
     what():  [05:42:13] /home/masa/projects/dev/tvm/src/runtime/cuda/cuda_device_api.cc:126: 
   ---------------------------------------------------------------
   An internal invariant was violated during the execution of TVM.
   Please read TVM's error reporting guidelines.
   More details can be found here: https://discuss.tvm.ai/t/error-reporting/7793.
   ---------------------------------------------------------------
     Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading == false: CUDA: an illegal memory access was encountered
   Stack trace:
     [bt] (0) /home/masa/projects/dev/tvm/build/libtvm.so(+0x14aa8e8) [0x7f4fcb8ca8e8]
     [bt] (1) /home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::CUDADeviceAPI::FreeDataSpace(DLContext, void*)+0xe4) [0x7f4fcb8cabe4]
     [bt] (2) /home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::NDArray::Internal::DefaultDeleter(tvm::runtime::Object*)+0x5b) [0x7f4fcb8593fb]
     [bt] (3) /home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::NDArray::CopyTo(DLContext const&) const+0x325) [0x7f4fcb5e4915]
     [bt] (4) /home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::CopyTo(tvm::runtime::ObjectRef, DLContext const&)+0x311) [0x7f4fcb884b11]
     [bt] (5) /home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::VirtualMachine::RunLoop()+0x2aee) [0x7f4fcb880dde]
     [bt] (6) /home/masa/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::VirtualMachine::Invoke(tvm::runtime::vm::VMFunction const&, std::vector<tvm::runtime::ObjectRef, std::allocator<tvm::runtime::ObjectRef> > const&)+0x27) [0x7f4fcb881c17]
     [bt] (7) /home/masa/projects/dev/tvm/build/libtvm.so(+0x14621f0) [0x7f4fcb8821f0]
     [bt] (8) /home/masa/projects/dev/tvm/build/libtvm.so(TVMFuncCall+0x63) [0x7f4fcb835613]
   ```
   
   @trevor-m Are you sure this is caused by `get_valid_counts` change? I've also changed NMS in https://github.com/apache/tvm/pull/7172,  I hope that change is fine.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org