MXNet Thread Safety for Inference

[Naveen Swamy <mn...@gmail.com>]

I re-opened this issue that bulk closed a few months ago
https://github.com/apache/incubator-mxnet/issues/3946 and I think issue
still remains is a blocking one for users.

From the conversation I gather that MXNet expects that only 1 thread
communicate with an executor. This would serialize all calls through the
thread and could lead to starvation if this thread does not get scheduled.
It also is a problem if the thread dies which requires you to restart the
process.

One of the proposed solution in the issue of forking the process and
relying on COW may not be feasible and even cumbersome to implement for
some use-cases such as used with JVM environments which are multi-threaded.
I think this quite a big restriction for many use-cases and believe this
issue needs to be addressed. One such use-case is
https://discuss.mxnet.io/t/fixing-thread-safety-issues-in-scala-library/236

thoughts?

Thanks Naveen