[GitHub] [incubator-mxnet] anirudh2290 commented on issue #16431: [RFC] MXNet Multithreaded Inference Interface

[GitBox <gi...@apache.org>]

anirudh2290 commented on issue #16431: [RFC] MXNet Multithreaded Inference Interface
URL: https://github.com/apache/incubator-mxnet/issues/16431#issuecomment-540834971
 
 
   Thanks @marcoabreu ! 
   
   > Will the new C-API functions be threadsafe in general? Speak, I can invoke them at any point in time from any thread without the need of a lock, sticky-thread or a thread hierarchy? (I'm thinking of the thread-safety being done on the backend level)
   
   The issue I found with C API thread safety especially with the cached op use case was the ThreadLocalStore. If we fix this issue then C APIs related to CreateCachedOp and InvokeCachedOp should be threadsafe.
   
   >  Will this also support the GPU use-case? Speak, the parameters are only copied into GPU memory once in the same fashion as you're describing for the CPU?
   
   This should still support the single GPU use-case for 1.6. Multi GPU inference use case requires more verification at the cached op level .
   
   > Do you think there's a path forward to make all inference-related C-APIs threadsafe instead of splitting off another execution branch?
   
   I don't think we have such a strict split between inference and training APIs at the C API level. For example for gluon cached op we call InvokeCachedOp for both training and Inference.
   
   But if I rephrase your question to:
   Will I be able to do multi threaded inference from every frontend API which I can use to do inference today ?  
   Right now, I am targeting only gluon since most users have been directed towards gluon. The other ways are using module, symbolic and using C Predict API. To support these two frontend APIs requires the graph executor to be thread safe.  This would definitely be a great add for MXNet since it would ensure that they can do multi-threaded inference from any of these APIs in MXNet, but not something I have planned for currently.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services