[GitHub] [tvm] eleflea opened a new issue #7782: Floating point exception when deploying model with C++ API

[GitBox <gi...@apache.org>]

eleflea opened a new issue #7782:
URL: https://github.com/apache/tvm/issues/7782


   Hi.
   I try to deploy yolov5 model with C++ API. I have export a `.so` file with `lib.export_library`. Below is a part of the C++ code.
   
   ```C++
   class YoloDetection
   {
   public:
       YoloDetection() {}
       ~YoloDetection();
       YoloDetection(string, string);
       vector<Det> predict(const cv::Mat &);
       void load(string, string);
   
       float conf_thresh = 0.4;
       float nms_thresh = 0.6;
       int max_dets = 300;
       int max_wh = 4096;
   protected:
       string lib_path = "";
       DLContext device;
       DLTensor *input_tensor;
       tvm::runtime::Module handler;
   };
   
   YoloDetection::YoloDetection(string lib_path, string device):
       lib_path(lib_path)
   {
       this->device = device == "gpu" ? DLContext{kDLGPU, 0} : DLContext{kDLCPU, 0};
       tvm::runtime::Module lib = tvm::runtime::Module::LoadFromFile(lib_path);
       int64_t in_shape[4] = {1, 3, IN_SIZE_H, IN_SIZE_W};
       TVMArrayAlloc(in_shape, 4, kDLFloat, 32, 1, kDLGPU, 0, &this->input_tensor);
       this->handler = lib.GetFunction("default")(this->device);
   }
   
   vector<Det> YoloDetection::predict(const cv::Mat &image) {
       auto pimg = pre_process(image);
       auto set_input = handler.GetFunction("set_input");
       auto get_output = handler.GetFunction("get_output");
       auto run = handler.GetFunction("run");
       TVMArrayCopyFromBytes(this->input_tensor, (void *)pimg.data(), IN_NUMEL*sizeof(float));
       set_input("input", this->input_tensor);
       run();
       tvm::runtime::NDArray out = get_output(0);
       size_t s = 1;
       for (auto i = 0; i < out->ndim; ++i)
       {
           s *= out->shape[i];
       }
       if (this->device.device_type != kDLCPU) {
           out = out.CopyTo(DLContext{kDLCPU, 0});
       }
       auto dets = post_process(static_cast<float *>(out->data), s, this->conf_thresh, this->nms_thresh, image.size());
       return dets;
   }
   ```
   
   But when I run this code, it shows:
   
   ```
   Floating point exception (core dumped)
   ```
   
   I found that error was in `run();` line.
   
   I moved `lib.GetFunction("default")(this->device)` to `predict` method. The program is fine, but got a time overhead due to that function call (for ~26.5ms I believe).
   
   So how to avoid GetFunction of default in predict, or there is other best practices?
   Thank you very much!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] eleflea closed issue #7782: Floating point exception when deploying model with C++ API

[GitBox <gi...@apache.org>]

eleflea closed issue #7782:
URL: https://github.com/apache/tvm/issues/7782


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] eleflea commented on issue #7782: Floating point exception when deploying model with C++ API

[GitBox <gi...@apache.org>]

eleflea commented on issue #7782:
URL: https://github.com/apache/tvm/issues/7782#issuecomment-812401353


   Don't know why but i solved it. We should hold both `lib_factory` and `gmod`. Which means:
   
   ```C++
   class YoloDetection
   {
   public:
       YoloDetection() {}
       ~YoloDetection();
       YoloDetection(string, string);
       vector<Det> predict(const cv::Mat &);
       void load(string, string);
   
       float conf_thresh = 0.4;
       float nms_thresh = 0.6;
       int max_dets = 300;
       int max_wh = 4096;
   
   protected:
       string lib_path = "";
       DLContext device;
       DLTensor *input_tensor = nullptr;
       tvm::runtime::Module lib_factory;
       tvm::runtime::Module gmod;
   };
   
   YoloDetection::~YoloDetection()
   {
       if (input_tensor != nullptr)
           TVMArrayFree(input_tensor);
   }
   
   YoloDetection::YoloDetection(string lib_path, string device) : lib_path(lib_path)
   {
       this->device = device == "gpu" ? DLContext{kDLGPU, 0} : DLContext{kDLCPU, 0};
       lib_factory = tvm::runtime::Module::LoadFromFile(lib_path);
       gmod = lib_factory.GetFunction("default")(this->device);
       int64_t in_shape[4] = {1, 3, IN_SIZE_H, IN_SIZE_W};
       TVMArrayAlloc(in_shape, 4, kDLFloat, 32, 1, kDLGPU, 0, &input_tensor);
   }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org