[GitHub] [tvm] masahi commented on pull request #8285: [VM][PooledAllocator] try reallocation once when OOM

[GitBox <gi...@apache.org>]

masahi commented on pull request #8285:
URL: https://github.com/apache/tvm/pull/8285#issuecomment-903641187


   This change doesn't solve the issue in https://github.com/apache/tvm/issues/8233, because `AllocDataSpace` can be called from `NDArray::Empty`: https://github.com/apache/tvm/blob/4d9bc9b4a3e9e8d3420efe60a52964fcd4c29c8d/src/runtime/ndarray.cc#L196-L197
   
   That call is not protected by try/catch, so if almost all memory are held by `PooledAllocator` and `NDArray::Empty` is called, the program crashes with the following error:
   ```
   terminate called after throwing an instance of 'tvm::runtime::InternalError'
     what():  [19:12:54] /home/masa/projects/dev/tvm/src/runtime/vulkan/vulkan_stream.cc:123: 
   ---------------------------------------------------------------
   An error occurred during the execution of TVM.
   For more information, please see: https://tvm.apache.org/docs/errors.html
   ---------------------------------------------------------------
     Check failed: (__e == VK_SUCCESS) is false: Vulkan Error, code=-13: Unknown Vulkan error code
   Stack trace:
     0: tvm::runtime::vulkan::VulkanStream::Synchronize()
     1: _ZN3tvm7runtime6vulkan15VulkanDeviceAPI13FreeDataSpac
     2: tvm::runtime::NDArray::Internal::DefaultDeleter(tvm::runtime::Object*)
     3: tvm::runtime::NDArray::CopyTo(DLDevice const&) const
     4: tvm::runtime::vm::CopyTo(tvm::runtime::ObjectRef, DLDevice const&)
     5: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::vm::VirtualMachine::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::$_6>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
     6: TVMFuncCall
   ``` 
   
   I think we need to revisit the memory release strategy of `PooledAllocator`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org