[GitHub] zheng-da commented on a change in pull request #12456: [MXNET-910] Multithreading inference.

[GitBox <gi...@apache.org>]

zheng-da commented on a change in pull request #12456: [MXNET-910] Multithreading inference.
URL: https://github.com/apache/incubator-mxnet/pull/12456#discussion_r218632768
 
 

 ##########
 File path: src/c_api/c_predict_api.cc
 ##########
 @@ -232,24 +223,117 @@ int MXPredCreatePartialOut(const char* symbol_json_str,
     }
     aux_arrays.push_back(nd);
   }
-  ret->arg_arrays = arg_arrays;
-  ret->aux_arrays = aux_arrays;
   // bind
-  {
-    std::map<std::string, Context> ctx_map;
-    std::vector<NDArray> grad_store(arg_arrays.size());
-    std::vector<OpReqType> grad_req(arg_arrays.size(), kNullOp);
-
-
-    ret->exec.reset(Executor::Bind(sym, ctx, ctx_map,
-                                   arg_arrays,
-                                   grad_store, grad_req,
-                                   aux_arrays));
+  for (int i = 0; i < num_threads; i++) {
+    std::unique_ptr<MXAPIPredictor> ret(new MXAPIPredictor());
+    ret->sym = sym;
+    ret->ctx = ctx;
+    ret->key2arg = key2arg;
+    ret->arg_arrays = arg_arrays;
+    ret->aux_arrays = aux_arrays;
     ret->out_shapes = out_shapes;
-    ret->out_arrays = ret->exec->outputs();
+
+    if (!lazy) {
 
 Review comment:
   The fundamental problem here is that if we create multiple executors in the same thread (e.g., in the main thread), these executors will share the same temporary resources, which leads to race condition when these executors are used in different threads. To fix this problem, here we avoid creating executors when we create predictors in the main thread. The executors are actually created when the predictor is used in the worker thread for the first time. As long as the executor is always used in this worker thread, there won't be race condition.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services