[GitHub] [tvm] srkreddy1238 opened a new pull request, #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

srkreddy1238 opened a new pull request, #13791:
URL: https://github.com/apache/tvm/pull/13791

   This application helps as a reference for verifying and integration of TVM compiled models on Android targets natively independent of RPC setup.
   
   tvmc will be used to for compiling tuning and to test run it before deployment.
   
   This PR also covers
    * Enabling clml for tvmc compilation tool.
    * Graph runtime api "get_output_info" to return output tensor specification similar to "get_input_into"
    * This tool adds and enabled 3rdparty dependency "cnpy" to deal with npz files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] tvm-bot commented on pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

tvm-bot commented on PR #13791:
URL: https://github.com/apache/tvm/pull/13791#issuecomment-1383526361

   <!---bot-comment-->
   
   Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from [Reviewers](https://github.com/apache/incubator-tvm/blob/master/CONTRIBUTORS.md#reviewers) by @-ing them in a comment.
   
   <!--bot-comment-ccs-start-->
    * No users to tag found in teams: `tool`, `native` <sub>See [#10317](https://github.com/apache/tvm/issues/10317) for details</sub><!--bot-comment-ccs-end-->
   
   <sub>Generated by [tvm-bot](https://github.com/apache/tvm/blob/main/ci/README.md#github-actions)</sub>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] srkreddy1238 commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

srkreddy1238 commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1073225743


##########
apps/cpp_rtvm/tvm_runner.cc:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm_runner.cc
+ * \brief TVM model runner implementation.
+ */
+
+#include "tvm_runner.h"
+
+#include <cnpy.h>
+
+#include <fstream>
+#include <streambuf>
+#include <string>
+
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief Get the TVM device id corresponding to device string.
+ * \param device the target device in string format.
+ * \return cl_device corresponding to the device string.
+ */
+int GetTVMDevice(std::string device) {
+  if (!device.compare("cl")) {
+    return static_cast<int>(kDLOpenCL);
+  } else {
+    LOG(FATAL) << "TVMRunner : Unsupported device :" << device;
+  }
+}
+
+/*!
+ * \brief Constructor for TVMRunner.
+ * \param path where the tfm compiler artifacts present.
+ * \param device the target device where we need to load the compiled model.
+ */
+TVMRunner::TVMRunner(std::string path, std::string device) : r_model_path(path), r_device(device) {
+  LOG(INFO) << "TVMRunner Constructor:" << r_model_path << " Devices:" << r_device;
+}
+
+/*!
+ * \brief Load Setup TVM graph runtime for given model.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::Load(void) {
+  LOG(INFO) << "TVMRunner Load:" << r_model_path;
+  // Load the lib file
+  r_mod_handle = Module::LoadFromFile((r_model_path + "/mod.so").c_str(), "so");
+
+  // Read model json file
+  std::ifstream json_reader((r_model_path + "/mod.json").c_str());
+  CHECK(!json_reader.fail()) << "Failed to open json file:" << (r_model_path + "/mod.json").c_str();
+  std::string json_str((std::istreambuf_iterator<char>(json_reader)),
+                       std::istreambuf_iterator<char>());
+  json_reader.close();
+
+  // Get ref to graph exeutor
+  auto f_handle = tvm::runtime::Registry::Get("tvm.graph_executor.create");
+
+  // Greate graph runtime
+  r_graph_handle = (*f_handle)(json_str, r_mod_handle, GetTVMDevice(r_device), 0);
+
+  // Read params binary file
+  std::ifstream params_reader((r_model_path + "/mod.params").c_str(), std::ios::binary);
+  CHECK(!params_reader.fail()) << "Failed to open json file:"
+                               << (r_model_path + "/mod.params").c_str();
+  const std::string params_str((std::istreambuf_iterator<char>(params_reader)),
+                               std::istreambuf_iterator<char>());
+  params_reader.close();
+  TVMByteArray params_arr;
+  params_arr.data = params_str.c_str();
+  params_arr.size = params_str.length();
+
+  // Load parameters
+  r_graph_handle.GetFunction("load_params")(params_arr);
+
+  return 0;
+}
+
+/*!
+ * \brief Calculated the memory size for the NDArray.
+ * \param NDArray object.
+ * \return size of the memory.
+ */
+inline size_t GetMemSize(NDArray& narr) {
+  size_t size = 1;
+  for (tvm_index_t i = 0; i < narr->ndim; ++i) {
+    size *= static_cast<size_t>(narr->shape[i]);
+  }
+  size *= (narr->dtype.bits * narr->dtype.lanes + 7) / 8;
+  return size;
+}
+
+/*!
+ * \brief Get the input alloc mem size.
+ * \param input_id The input id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetInputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetInputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Get the output alloc mem size.
+ * \param input_id The output id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetOutputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetOutputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_output")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Set the model inputs from npz file.
+ * \param inputfile the npz file from where we read input tensor data.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string inputfile) {
+  LOG(INFO) << "TVMRunner::SetInput (Numpy):" << inputfile;
+  cnpy::npz_t npz_input = cnpy::npz_load(inputfile);
+
+  for (auto& elem : mInfo.input_info) {
+    LOG(INFO) << "Set Numpy Input for :" << elem.first;
+    NDArray in_arr = r_graph_handle.GetFunction("get_input")(elem.first);
+    auto ssize = GetMemSize(in_arr);
+
+    if (npz_input.find(elem.first) != npz_input.end()) {
+      in_arr.CopyFromBytes(npz_input[elem.first].data<char>(), ssize);
+    } else {
+      LOG(WARNING) << "Couldn't find input " << elem.first << " in npy input file";
+    }
+  }
+
+  return 0;
+}
+
+/*!
+ * \brief Set the model input from the given binary buffer.
+ * \param input_id input node name.
+ * \param raw_input binary input buffer to copy over input NDArray.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string input_id, char* raw_input) {
+  LOG(INFO) << "TVMRunner::SetInput (Raw)";
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);

Review Comment:
   But, I need in_arr for data copy below.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] tqchen commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

tqchen commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1073772649


##########
.gitmodules:
##########
@@ -19,3 +19,6 @@
 [submodule "3rdparty/OpenCL-Headers"]
 	path = 3rdparty/OpenCL-Headers
 	url = https://github.com/KhronosGroup/OpenCL-Headers.git
+[submodule "3rdparty/cnpy"]
+	path = 3rdparty/cnpy
+	url = https://github.com/rogersce/cnpy.git

Review Comment:
   Personally I still think having an util to be able to create such a dump can be helpful. Mainly because the numpy format can be complicated and there will be questions of supporting other formats, such as protobuf. So converting to a standard one that we commonly use can be better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] junrushao commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

junrushao commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1071822320


##########
.gitmodules:
##########
@@ -19,3 +19,6 @@
 [submodule "3rdparty/OpenCL-Headers"]
 	path = 3rdparty/OpenCL-Headers
 	url = https://github.com/KhronosGroup/OpenCL-Headers.git
+[submodule "3rdparty/cnpy"]
+	path = 3rdparty/cnpy
+	url = https://github.com/rogersce/cnpy.git

Review Comment:
   Thanks for the context! If so, I do not object the idea as long as it's not part of default build



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] echuraev commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

echuraev commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1082147944


##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,354 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using `rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py has a reference to prepare sample network ```resnet50``` from keras framework.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section [RPC setup](#rpc-setup) for the same.
+
+Auto tunng is necessary to obtain best performaning kernels. We can skip this step if we have tuning log already or the tuning cache is available from tophub (implicite by TVM compilation process).
+Below message indicate that there exists some kernels not optimized for the selected target. In this case we can proceed with tuning to best performance.
+```One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.```
+
+with below environment from [RPC setup](#rpc-setup)
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+the model to be tuned being ```./model_data/keras-resnet50/resnet50.h5```
+```
+
+the below command we can generate the tuning cache to file ```./model_data/keras-resnet50/keras-resnet50.log```
+
+```bash
+python3 -m tvm.driver.tvmc tune --target="opencl" --target-host="llvm -mtriple=aarch64-linux-gnu" \
+./model_data/keras-resnet50/resnet50.h5 -o ./model_data/keras-resnet50/keras-resnet50.log \
+--early-stopping 0 --repeat 30 --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --trials 1024 \
+--tuning-records ./model_data/keras-resnet50/keras-resnet50-records.log --tuner xgb
+```
+
+where
+```bash
+--target="opencl" refers to opencl device on Android device
+--target-host="llvm -mtriple=aarch64-linux-gnu" refers to target_host being an ARM64 CPU
+Options --early-stopping, --repeat, --trials, --tuner are Auto TVM specific options.
+```
+Please refer to AutoTVM documentation for more details [here](https://tvm.apache.org/docs/how_to/tune_with_autotvm/index.html?highlight=autotvm).
+
+### Compile the model
+
+Compilation step generates TVM compiler output artifacts which need to be taken to target device for deployment.
+These artifacts is a compressed archive with kernel shared lib, json with graph description and params binary.
+
+Below command will generate the same
+
+
+```bash
+python3 -m tvm.driver.tvmc compile --cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang \
+--target="opencl, llvm" --target-llvm-mtriple aarch64-linux-gnu -o keras-resnet50.tar ./model_data/keras-resnet50/resnet50.h5
+```
+
+where
+```
+--cross-compiler : Indicates the cross compiler path for kernel library generation
+--target="opencl, llvm" indicates target and host devices
+```
+
+### Test Run via RPC
+
+At this stage we can verify the generated compiler output for execution correctness over the RPC setup interface.
+Below command can run the compiled output on remote target device.
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+compilation out being keras-resnet50.tar
+```
+
+```bash
+python3 -m tvm.driver.tvmc run --device="opencl" keras-resnet50.tar --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --print-time
+```
+
+This inputs random inputs and validates the execution correctness of the compiled model.
+
+```tvmc``` tool has various options to input custom data, profile the model and benchmark the execution.
+
+
+### Deployment Run
+
+Now we will verify the deployment run of the compiled model using ```rtvm``` tool on target device without any RPC or host based execution.
+
+We need to extract the tar achive on target device. We can copy the extracted contents of ```keras-resnet50.tar``` under Android temp folder at ```/data/local/tmp/keras-resnet50/```
+
+Also copy the cross compiled tool ```rtvm``` and ```libtvm_runtime.so``` to ```data/local/tmp/```
+
+```rtvm``` usage can be quired as below
+```bash
+Android:/data/local/tmp $ LD_LIBRARY_PATH=./ ./rtvm
+Command line usage
+--model        - The folder containing tvm artifacts(mod.so, mod.param, mod.json)
+--device       - The target device to use {llvm, opencl, cpu, cuda, metal, rocm, vpi, oneapi}
+--input        - Numpy file for the model input (optional and we use random of not given)
+--output       - Numpy file name to dump the model output as numpy
+--dump-meta    - Dump model meta information
+
+  Example
+  ./rtvm --model=keras-resnet50 --device="opencl" --dump-meta
+  ./rtvm --model=keras-resnet50 --device="opencl" --input input.npz --output=output.npz
+```
+
+```rtvm``` can run the model using no inputs (just a dry run without any valid inputs) and also with specific input supplied as a numpy npz format file.
+
+We can create npz dump for all inputs by saving the dict object as hown below.
+
+With ```keras-resnet50``` having one  input ```input_1``` with shape ```[1, 224, 224, 3]``` and dtype ```float32```
+
+```
+# Random initilization
+input1 = np.random.uniform(low=-1, high=1, size=(1, 224, 224, 3)).astype("float32")
+dataset = {"input_1": input1}
+np.savez("input.npz", **dataset)
+```
+
+Copy ```input.npz``` also to the target device as ```/data/local/tmp/input.npz```
+
+
+Now, on Android shell we can do a dry run as well as with specific input as shown below.
+```bash
+# Query meta data information
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl" --dump-meta
+. . . . . .
+Meta Information:keras-resnet50
+    Number of Inputs:183
+    Number of Outputs:1
+    Input MetaInfo:
+        Input:input_1
+            DType:float32
+            Shape:[1, 224, 224, 3]
+    Output MetaInfo:
+        Output:tvmgen_default_fused_nn_softmax
+            DType:float32
+            Shape:[1, 1000]
+. . . . . .
+
+# Dry run with out any inputs
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl"
+Model         = keras-resnet50
+Device        = opencl
+Input         =
+Output        =
+Dump Metadata = False
+TVMRunner Constructor:keras-resnet50 Devices:opencl
+TVMRunner Load:keras-resnet50
+TVMRunner::GetMetaInfo
+Executing dry run ...
+Set Random Input for :input_1
+TVMRunner::GetInputMemSize:input_1
+Random Input Size:602112  bytes
+TVMRunner::SetInput (Raw)
+TVMRunner::Run
+Get Output for :tvmgen_default_fused_nn_softmax
+TVMRunner::GetOutputMemSize:tvmgen_default_fused_nn_softmax
+TVMRunner::GetOutput (Raw)
+Output Size:4000  bytes
+
+
+# Run with input and dump output as npz file
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl"

Review Comment:
   There is no difference with the previous command. Should we pass parameters `--input` and `--output`?



##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,354 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using `rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py has a reference to prepare sample network ```resnet50``` from keras framework.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section [RPC setup](#rpc-setup) for the same.
+
+Auto tunng is necessary to obtain best performaning kernels. We can skip this step if we have tuning log already or the tuning cache is available from tophub (implicite by TVM compilation process).
+Below message indicate that there exists some kernels not optimized for the selected target. In this case we can proceed with tuning to best performance.
+```One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.```
+
+with below environment from [RPC setup](#rpc-setup)
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+the model to be tuned being ```./model_data/keras-resnet50/resnet50.h5```
+```
+
+the below command we can generate the tuning cache to file ```./model_data/keras-resnet50/keras-resnet50.log```
+
+```bash
+python3 -m tvm.driver.tvmc tune --target="opencl" --target-host="llvm -mtriple=aarch64-linux-gnu" \
+./model_data/keras-resnet50/resnet50.h5 -o ./model_data/keras-resnet50/keras-resnet50.log \
+--early-stopping 0 --repeat 30 --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --trials 1024 \
+--tuning-records ./model_data/keras-resnet50/keras-resnet50-records.log --tuner xgb
+```
+
+where
+```bash
+--target="opencl" refers to opencl device on Android device
+--target-host="llvm -mtriple=aarch64-linux-gnu" refers to target_host being an ARM64 CPU
+Options --early-stopping, --repeat, --trials, --tuner are Auto TVM specific options.
+```
+Please refer to AutoTVM documentation for more details [here](https://tvm.apache.org/docs/how_to/tune_with_autotvm/index.html?highlight=autotvm).
+
+### Compile the model
+
+Compilation step generates TVM compiler output artifacts which need to be taken to target device for deployment.
+These artifacts is a compressed archive with kernel shared lib, json with graph description and params binary.
+
+Below command will generate the same
+
+
+```bash
+python3 -m tvm.driver.tvmc compile --cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang \
+--target="opencl, llvm" --target-llvm-mtriple aarch64-linux-gnu -o keras-resnet50.tar ./model_data/keras-resnet50/resnet50.h5
+```
+
+where
+```
+--cross-compiler : Indicates the cross compiler path for kernel library generation
+--target="opencl, llvm" indicates target and host devices
+```
+
+### Test Run via RPC
+
+At this stage we can verify the generated compiler output for execution correctness over the RPC setup interface.
+Below command can run the compiled output on remote target device.
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+compilation out being keras-resnet50.tar
+```
+
+```bash
+python3 -m tvm.driver.tvmc run --device="opencl" keras-resnet50.tar --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --print-time
+```
+
+This inputs random inputs and validates the execution correctness of the compiled model.
+
+```tvmc``` tool has various options to input custom data, profile the model and benchmark the execution.
+
+
+### Deployment Run
+
+Now we will verify the deployment run of the compiled model using ```rtvm``` tool on target device without any RPC or host based execution.
+
+We need to extract the tar achive on target device. We can copy the extracted contents of ```keras-resnet50.tar``` under Android temp folder at ```/data/local/tmp/keras-resnet50/```
+
+Also copy the cross compiled tool ```rtvm``` and ```libtvm_runtime.so``` to ```data/local/tmp/```
+
+```rtvm``` usage can be quired as below
+```bash
+Android:/data/local/tmp $ LD_LIBRARY_PATH=./ ./rtvm
+Command line usage
+--model        - The folder containing tvm artifacts(mod.so, mod.param, mod.json)
+--device       - The target device to use {llvm, opencl, cpu, cuda, metal, rocm, vpi, oneapi}
+--input        - Numpy file for the model input (optional and we use random of not given)
+--output       - Numpy file name to dump the model output as numpy
+--dump-meta    - Dump model meta information
+
+  Example
+  ./rtvm --model=keras-resnet50 --device="opencl" --dump-meta
+  ./rtvm --model=keras-resnet50 --device="opencl" --input input.npz --output=output.npz
+```
+
+```rtvm``` can run the model using no inputs (just a dry run without any valid inputs) and also with specific input supplied as a numpy npz format file.
+
+We can create npz dump for all inputs by saving the dict object as hown below.

Review Comment:
   ```suggestion
   We can create npz dump for all inputs by saving the dict object as shown below.
   ```



##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,354 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using `rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py has a reference to prepare sample network ```resnet50``` from keras framework.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section [RPC setup](#rpc-setup) for the same.
+
+Auto tunng is necessary to obtain best performaning kernels. We can skip this step if we have tuning log already or the tuning cache is available from tophub (implicite by TVM compilation process).
+Below message indicate that there exists some kernels not optimized for the selected target. In this case we can proceed with tuning to best performance.
+```One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.```
+
+with below environment from [RPC setup](#rpc-setup)
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+the model to be tuned being ```./model_data/keras-resnet50/resnet50.h5```
+```
+
+the below command we can generate the tuning cache to file ```./model_data/keras-resnet50/keras-resnet50.log```
+
+```bash
+python3 -m tvm.driver.tvmc tune --target="opencl" --target-host="llvm -mtriple=aarch64-linux-gnu" \
+./model_data/keras-resnet50/resnet50.h5 -o ./model_data/keras-resnet50/keras-resnet50.log \
+--early-stopping 0 --repeat 30 --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --trials 1024 \
+--tuning-records ./model_data/keras-resnet50/keras-resnet50-records.log --tuner xgb
+```
+
+where
+```bash
+--target="opencl" refers to opencl device on Android device
+--target-host="llvm -mtriple=aarch64-linux-gnu" refers to target_host being an ARM64 CPU
+Options --early-stopping, --repeat, --trials, --tuner are Auto TVM specific options.
+```
+Please refer to AutoTVM documentation for more details [here](https://tvm.apache.org/docs/how_to/tune_with_autotvm/index.html?highlight=autotvm).
+
+### Compile the model
+
+Compilation step generates TVM compiler output artifacts which need to be taken to target device for deployment.
+These artifacts is a compressed archive with kernel shared lib, json with graph description and params binary.
+
+Below command will generate the same
+
+
+```bash
+python3 -m tvm.driver.tvmc compile --cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang \
+--target="opencl, llvm" --target-llvm-mtriple aarch64-linux-gnu -o keras-resnet50.tar ./model_data/keras-resnet50/resnet50.h5
+```
+
+where
+```
+--cross-compiler : Indicates the cross compiler path for kernel library generation
+--target="opencl, llvm" indicates target and host devices
+```
+
+### Test Run via RPC
+
+At this stage we can verify the generated compiler output for execution correctness over the RPC setup interface.
+Below command can run the compiled output on remote target device.
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+compilation out being keras-resnet50.tar
+```
+
+```bash
+python3 -m tvm.driver.tvmc run --device="opencl" keras-resnet50.tar --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --print-time
+```
+
+This inputs random inputs and validates the execution correctness of the compiled model.
+
+```tvmc``` tool has various options to input custom data, profile the model and benchmark the execution.
+
+
+### Deployment Run
+
+Now we will verify the deployment run of the compiled model using ```rtvm``` tool on target device without any RPC or host based execution.
+
+We need to extract the tar achive on target device. We can copy the extracted contents of ```keras-resnet50.tar``` under Android temp folder at ```/data/local/tmp/keras-resnet50/```
+
+Also copy the cross compiled tool ```rtvm``` and ```libtvm_runtime.so``` to ```data/local/tmp/```
+
+```rtvm``` usage can be quired as below
+```bash
+Android:/data/local/tmp $ LD_LIBRARY_PATH=./ ./rtvm
+Command line usage
+--model        - The folder containing tvm artifacts(mod.so, mod.param, mod.json)
+--device       - The target device to use {llvm, opencl, cpu, cuda, metal, rocm, vpi, oneapi}
+--input        - Numpy file for the model input (optional and we use random of not given)
+--output       - Numpy file name to dump the model output as numpy
+--dump-meta    - Dump model meta information
+
+  Example
+  ./rtvm --model=keras-resnet50 --device="opencl" --dump-meta
+  ./rtvm --model=keras-resnet50 --device="opencl" --input input.npz --output=output.npz
+```
+
+```rtvm``` can run the model using no inputs (just a dry run without any valid inputs) and also with specific input supplied as a numpy npz format file.
+
+We can create npz dump for all inputs by saving the dict object as hown below.
+
+With ```keras-resnet50``` having one  input ```input_1``` with shape ```[1, 224, 224, 3]``` and dtype ```float32```
+
+```
+# Random initilization
+input1 = np.random.uniform(low=-1, high=1, size=(1, 224, 224, 3)).astype("float32")
+dataset = {"input_1": input1}
+np.savez("input.npz", **dataset)
+```
+
+Copy ```input.npz``` also to the target device as ```/data/local/tmp/input.npz```
+
+
+Now, on Android shell we can do a dry run as well as with specific input as shown below.
+```bash
+# Query meta data information
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl" --dump-meta
+. . . . . .
+Meta Information:keras-resnet50
+    Number of Inputs:183
+    Number of Outputs:1
+    Input MetaInfo:
+        Input:input_1
+            DType:float32
+            Shape:[1, 224, 224, 3]
+    Output MetaInfo:
+        Output:tvmgen_default_fused_nn_softmax
+            DType:float32
+            Shape:[1, 1000]
+. . . . . .
+
+# Dry run with out any inputs
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl"
+Model         = keras-resnet50
+Device        = opencl
+Input         =
+Output        =
+Dump Metadata = False
+TVMRunner Constructor:keras-resnet50 Devices:opencl
+TVMRunner Load:keras-resnet50
+TVMRunner::GetMetaInfo
+Executing dry run ...
+Set Random Input for :input_1
+TVMRunner::GetInputMemSize:input_1
+Random Input Size:602112  bytes
+TVMRunner::SetInput (Raw)
+TVMRunner::Run
+Get Output for :tvmgen_default_fused_nn_softmax
+TVMRunner::GetOutputMemSize:tvmgen_default_fused_nn_softmax
+TVMRunner::GetOutput (Raw)
+Output Size:4000  bytes
+
+
+# Run with input and dump output as npz file
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl"
+Model         = keras-resnet50
+Device        = opencl
+Input         = input.npz
+Output        = output.npz
+Dump Metadata = False
+TVMRunner Constructor:keras-resnet50 Devices:opencl
+TVMRunner Load:keras-resnet50
+TVMRunner::GetMetaInfo
+Executing with Input:input.npz Output:output.npz
+TVMRunner::SetInput (Numpy):input.npz
+Set Numpy Input for :input_1
+TVMRunner::Run
+TVMRunner::GetOutput (Numpy):output.npz
+Get Output for :tvmgen_default_fused_nn_softmax
+Output Size:4000  bytes
+```
+
+output.npz contains the modle outputs. Below is a quick look of its contents.
+```bash
+tvm-host:~$ unzip -l output.npz
+Archive:  output.npz
+  Length      Date    Time    Name
+---------  ---------- -----   ----
+     4080  1980-00-00 00:00   tvmgen_default_fused_nn_softmax.npy
+---------                     -------
+     4080                     1 file
+
+```
+
+Building ```cpp_rtvm``` produces ```libtvm_runner.so```, a simplified interface that rtvm use internally for loading and executing tvm compiled models from C/C++ environments.
+```tvm_runner.h``` describes the interface definition here. Alternatively pro users can use TVM's [c_native_api](https://github.com/apache/tvm/blob/main/include/tvm/runtime/c_runtime_api.h) interface for more access to TVM features.
+
+
+# RPC Setup
+
+For Android devices we require cross compilation of tvm_rpc (also libtvm_runtime.so which is a dependency) for remote device.
+RPC setup involves running tracker on host device and running tvm_rpc on target device.
+
+### Tracker
+
+Below command runs the tracker on host over port ```9100```
+
+```bash
+python3 -m tvm.exec.rpc_tracker --host 127.0.0.1 --port 9100"
+```
+### RPC on Target
+
+With ```abcd1234ef``` being adb device id and tvm_rpc (and libtvm_runtime.so) is pushed to target device at ```/data/local/tmp/tvm_rpc/```
+
+```bash
+export ANDROID_SERIAL=abcd1234ef
+# Below settings will reroute networking tcm connections on devices to host device via adb interface
+adb reverse tcp:9100 tcp:9100
+adb forward tcp:5000 tcp:5000
+# Run the tvm_rpc on device
+env adb shell "cd /data/local/tmp/tvm_rpc; killall -9 tvm_rpc; \
+LD_LIBRARY_PATH=/data/local/tmp/tvm_rpc/ ./tvm_rpc server --host=0.0.0.0 --port=5000 --port-end=5010 --tracker=127.0.0.1:9100 --key=android
+```
+
+Now we have the rpc setup with ```TVM_TRACKER_HOST=127.0.0.1```, ```TVM_TRACKER_PORT=9100``` and ```TVM_RPC_KEY=android```.
+
+We can also check connected and available devices on tracker as shown below.
+
+```bash
+python3 -m tvm.exec.query_rpc_tracker --port ${TVM_TRACKER_PORT}
+Tracker address 127.0.0.1:9100
+
+Server List
+------------------------------
+server-address           key
+------------------------------
+       127.0.0.1:5000    server:android
+------------------------------
+
+Queue Status
+-------------------------------
+key       total  free  pending
+-------------------------------
+android   1      1     0
+-------------------------------
+```
+
+
+# Target Specific Configuration
+
+Below sections describe device/target specific settings to be used with ```tvmc``` and ```rtvm``` tools.

Review Comment:
   The instruction in this section is only related to the `tvmc`, isn't it?



##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,354 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using `rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py has a reference to prepare sample network ```resnet50``` from keras framework.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section [RPC setup](#rpc-setup) for the same.
+
+Auto tunng is necessary to obtain best performaning kernels. We can skip this step if we have tuning log already or the tuning cache is available from tophub (implicite by TVM compilation process).
+Below message indicate that there exists some kernels not optimized for the selected target. In this case we can proceed with tuning to best performance.
+```One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.```
+
+with below environment from [RPC setup](#rpc-setup)
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+the model to be tuned being ```./model_data/keras-resnet50/resnet50.h5```
+```
+
+the below command we can generate the tuning cache to file ```./model_data/keras-resnet50/keras-resnet50.log```
+
+```bash
+python3 -m tvm.driver.tvmc tune --target="opencl" --target-host="llvm -mtriple=aarch64-linux-gnu" \
+./model_data/keras-resnet50/resnet50.h5 -o ./model_data/keras-resnet50/keras-resnet50.log \
+--early-stopping 0 --repeat 30 --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --trials 1024 \
+--tuning-records ./model_data/keras-resnet50/keras-resnet50-records.log --tuner xgb
+```
+
+where
+```bash
+--target="opencl" refers to opencl device on Android device
+--target-host="llvm -mtriple=aarch64-linux-gnu" refers to target_host being an ARM64 CPU
+Options --early-stopping, --repeat, --trials, --tuner are Auto TVM specific options.
+```
+Please refer to AutoTVM documentation for more details [here](https://tvm.apache.org/docs/how_to/tune_with_autotvm/index.html?highlight=autotvm).
+
+### Compile the model
+
+Compilation step generates TVM compiler output artifacts which need to be taken to target device for deployment.
+These artifacts is a compressed archive with kernel shared lib, json with graph description and params binary.
+
+Below command will generate the same
+
+
+```bash
+python3 -m tvm.driver.tvmc compile --cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang \
+--target="opencl, llvm" --target-llvm-mtriple aarch64-linux-gnu -o keras-resnet50.tar ./model_data/keras-resnet50/resnet50.h5
+```
+
+where
+```
+--cross-compiler : Indicates the cross compiler path for kernel library generation
+--target="opencl, llvm" indicates target and host devices
+```
+
+### Test Run via RPC
+
+At this stage we can verify the generated compiler output for execution correctness over the RPC setup interface.
+Below command can run the compiled output on remote target device.
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+compilation out being keras-resnet50.tar
+```
+
+```bash
+python3 -m tvm.driver.tvmc run --device="opencl" keras-resnet50.tar --rpc-key ${TVM_RPC_KEY} --rpc-tracker ${TVM_TRACKER_HOST}:${TVM_TRACKER_PORT} --print-time
+```
+
+This inputs random inputs and validates the execution correctness of the compiled model.
+
+```tvmc``` tool has various options to input custom data, profile the model and benchmark the execution.
+
+
+### Deployment Run
+
+Now we will verify the deployment run of the compiled model using ```rtvm``` tool on target device without any RPC or host based execution.
+
+We need to extract the tar achive on target device. We can copy the extracted contents of ```keras-resnet50.tar``` under Android temp folder at ```/data/local/tmp/keras-resnet50/```
+
+Also copy the cross compiled tool ```rtvm``` and ```libtvm_runtime.so``` to ```data/local/tmp/```
+
+```rtvm``` usage can be quired as below
+```bash
+Android:/data/local/tmp $ LD_LIBRARY_PATH=./ ./rtvm
+Command line usage
+--model        - The folder containing tvm artifacts(mod.so, mod.param, mod.json)
+--device       - The target device to use {llvm, opencl, cpu, cuda, metal, rocm, vpi, oneapi}
+--input        - Numpy file for the model input (optional and we use random of not given)
+--output       - Numpy file name to dump the model output as numpy
+--dump-meta    - Dump model meta information
+
+  Example
+  ./rtvm --model=keras-resnet50 --device="opencl" --dump-meta
+  ./rtvm --model=keras-resnet50 --device="opencl" --input input.npz --output=output.npz
+```
+
+```rtvm``` can run the model using no inputs (just a dry run without any valid inputs) and also with specific input supplied as a numpy npz format file.
+
+We can create npz dump for all inputs by saving the dict object as hown below.
+
+With ```keras-resnet50``` having one  input ```input_1``` with shape ```[1, 224, 224, 3]``` and dtype ```float32```
+
+```
+# Random initilization
+input1 = np.random.uniform(low=-1, high=1, size=(1, 224, 224, 3)).astype("float32")
+dataset = {"input_1": input1}
+np.savez("input.npz", **dataset)
+```
+
+Copy ```input.npz``` also to the target device as ```/data/local/tmp/input.npz```
+
+
+Now, on Android shell we can do a dry run as well as with specific input as shown below.
+```bash
+# Query meta data information
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl" --dump-meta
+. . . . . .
+Meta Information:keras-resnet50
+    Number of Inputs:183
+    Number of Outputs:1
+    Input MetaInfo:
+        Input:input_1
+            DType:float32
+            Shape:[1, 224, 224, 3]
+    Output MetaInfo:
+        Output:tvmgen_default_fused_nn_softmax
+            DType:float32
+            Shape:[1, 1000]
+. . . . . .
+
+# Dry run with out any inputs
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl"
+Model         = keras-resnet50
+Device        = opencl
+Input         =
+Output        =
+Dump Metadata = False
+TVMRunner Constructor:keras-resnet50 Devices:opencl
+TVMRunner Load:keras-resnet50
+TVMRunner::GetMetaInfo
+Executing dry run ...
+Set Random Input for :input_1
+TVMRunner::GetInputMemSize:input_1
+Random Input Size:602112  bytes
+TVMRunner::SetInput (Raw)
+TVMRunner::Run
+Get Output for :tvmgen_default_fused_nn_softmax
+TVMRunner::GetOutputMemSize:tvmgen_default_fused_nn_softmax
+TVMRunner::GetOutput (Raw)
+Output Size:4000  bytes
+
+
+# Run with input and dump output as npz file
+Android:/data/local/tmp/ $ LD_LIBRARY_PATH=./ ./rtvm --model keras-resnet50 --device "opencl"
+Model         = keras-resnet50
+Device        = opencl
+Input         = input.npz
+Output        = output.npz
+Dump Metadata = False
+TVMRunner Constructor:keras-resnet50 Devices:opencl
+TVMRunner Load:keras-resnet50
+TVMRunner::GetMetaInfo
+Executing with Input:input.npz Output:output.npz
+TVMRunner::SetInput (Numpy):input.npz
+Set Numpy Input for :input_1
+TVMRunner::Run
+TVMRunner::GetOutput (Numpy):output.npz
+Get Output for :tvmgen_default_fused_nn_softmax
+Output Size:4000  bytes
+```
+
+output.npz contains the modle outputs. Below is a quick look of its contents.
+```bash
+tvm-host:~$ unzip -l output.npz
+Archive:  output.npz
+  Length      Date    Time    Name
+---------  ---------- -----   ----
+     4080  1980-00-00 00:00   tvmgen_default_fused_nn_softmax.npy
+---------                     -------
+     4080                     1 file
+
+```
+
+Building ```cpp_rtvm``` produces ```libtvm_runner.so```, a simplified interface that rtvm use internally for loading and executing tvm compiled models from C/C++ environments.
+```tvm_runner.h``` describes the interface definition here. Alternatively pro users can use TVM's [c_native_api](https://github.com/apache/tvm/blob/main/include/tvm/runtime/c_runtime_api.h) interface for more access to TVM features.
+
+
+# RPC Setup
+
+For Android devices we require cross compilation of tvm_rpc (also libtvm_runtime.so which is a dependency) for remote device.
+RPC setup involves running tracker on host device and running tvm_rpc on target device.
+
+### Tracker
+
+Below command runs the tracker on host over port ```9100```
+
+```bash
+python3 -m tvm.exec.rpc_tracker --host 127.0.0.1 --port 9100"
+```
+### RPC on Target
+
+With ```abcd1234ef``` being adb device id and tvm_rpc (and libtvm_runtime.so) is pushed to target device at ```/data/local/tmp/tvm_rpc/```
+
+```bash
+export ANDROID_SERIAL=abcd1234ef
+# Below settings will reroute networking tcm connections on devices to host device via adb interface
+adb reverse tcp:9100 tcp:9100
+adb forward tcp:5000 tcp:5000
+# Run the tvm_rpc on device
+env adb shell "cd /data/local/tmp/tvm_rpc; killall -9 tvm_rpc; \
+LD_LIBRARY_PATH=/data/local/tmp/tvm_rpc/ ./tvm_rpc server --host=0.0.0.0 --port=5000 --port-end=5010 --tracker=127.0.0.1:9100 --key=android
+```
+
+Now we have the rpc setup with ```TVM_TRACKER_HOST=127.0.0.1```, ```TVM_TRACKER_PORT=9100``` and ```TVM_RPC_KEY=android```.
+
+We can also check connected and available devices on tracker as shown below.
+
+```bash
+python3 -m tvm.exec.query_rpc_tracker --port ${TVM_TRACKER_PORT}
+Tracker address 127.0.0.1:9100
+
+Server List
+------------------------------
+server-address           key
+------------------------------
+       127.0.0.1:5000    server:android
+------------------------------
+
+Queue Status
+-------------------------------
+key       total  free  pending
+-------------------------------
+android   1      1     0
+-------------------------------
+```
+
+
+# Target Specific Configuration
+
+Below sections describe device/target specific settings to be used with ```tvmc``` and ```rtvm``` tools.
+
+### Adreno GPU
+
+Adreno GPU has a docker definition that helps to ease the development environment.
+
+We can build the docker image by using below command from TVM repo.
+
+```bash
+./docker/build.sh ci_adreno
+docker tag tvm.ci_adreno ci_adreno
+```
+
+Below command builds host and target rpc components for Adreno and drops into an interactive shell.
+
+```bash
+./tests/scripts/ci.py adreno -i
+```
+
+Also, one can build with Adreno OpenCLML SDK support
+
+```bash
+export ADRENO_OPENCL=<Path to OpenCLML SDK>
+./tests/scripts/ci.py adreno -i
+```
+
+Above command produces
+```build-adreno``` which is host build
+```build-adreno-target``` which contains cross compiled tvm_rpc and libtvm_runtime.so
+
+
+Below options to be used for Adreno GPU while working with tvmc
+
+* Tuning
+
+```
+--target="opencl -device=adreno"
+--target-host="llvm -mtriple=aarch64-linux-gnu"
+```
+
+* Compilation
+
+```
+--cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang
+--target="opencl, llvm"
+--target-opencl-device adreno
+--target-llvm-mtriple aarch64-linux-gnu
+```
+
+While enabling CLML just need to specify below target option for compilation.

Review Comment:
   Please, add indent in 2 spaces for all pieces of code and information which should be under the bullet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] srkreddy1238 commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

srkreddy1238 commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1073227619


##########
apps/cpp_rtvm/main.cc:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file main.cc
+ * \brief TVM runtime utility for TVM.
+ */
+#include <csignal>
+#include <cstdio>
+#include <cstdlib>
+#if defined(__linux__) || defined(__ANDROID__)
+#include <unistd.h>
+#endif
+#include <dmlc/logging.h>
+
+#include <cstring>
+#include <iostream>
+#include <sstream>
+#include <vector>
+
+#include "../../src/support/socket.h"
+#include "../../src/support/utils.h"
+#include "tvm_runner.h"
+
+#if defined(_WIN32)
+#include "win32_process.h"
+#endif
+
+using namespace std;
+using namespace tvm::runtime;
+using namespace tvm::support;
+
+static const string kUsage =
+    "Command line usage\n"
+    "--model        - The tvm artifact to load\n"
+    "--device       - The target device to use {llvm, cl, ...etc.}\n"
+    "--input        - Numpy file for the model input (optional and we use random of not given)\n"
+    "--output       - Numpy file name to dump the model output as numpy\n"
+    "--dump-meta    - Dump model meta information\n"
+    "\n"
+    "  Example\n"
+    "  ./rtvm --model keras-resnet50 --device=\"cl --dump-mata\"\n"
+    "  ./rtvm --model keras-resnet50 --device=\"cl\" --input input.npz --output=output.npz\n"
+    "\n";
+
+/*!
+ * \brief Tool Arguments.
+ * \arg model The tvm artifact to load & run
+ * \arg device The target device to use {llvm, cl, ...etc.}
+ * \arg input Numpy file for the model input
+ * \arg output Numpy file name to dump the model output as numpy
+ */
+struct ToolArgs {
+  string model;
+  string device;
+  string input;
+  string output;
+  bool dump_meta = false;
+};
+
+/*!
+ * \brief PrintArgs print the contents of ToolArgs
+ * \param args ToolArgs structure
+ */
+void PrintArgs(const ToolArgs& args) {
+  LOG(INFO) << "Model         = " << args.model;
+  LOG(INFO) << "Device        = " << args.device;
+  LOG(INFO) << "Input         = " << args.input;
+  LOG(INFO) << "Output        = " << args.output;
+  LOG(INFO) << "Dump Metadata = " << ((args.dump_meta) ? ("True") : ("False"));
+}
+
+#if defined(__linux__) || defined(__ANDROID__)
+/*!
+ * \brief CtrlCHandler, exits if Ctrl+C is pressed
+ * \param s signal
+ */
+void CtrlCHandler(int s) {
+  LOG(INFO) << "\nUser pressed Ctrl+C, Exiting";
+  exit(1);
+}
+
+/*!
+ * \brief HandleCtrlC Register for handling Ctrl+C event.
+ */
+void HandleCtrlC() {
+  // Ctrl+C handler
+  struct sigaction sigIntHandler;
+  sigIntHandler.sa_handler = CtrlCHandler;
+  sigemptyset(&sigIntHandler.sa_mask);
+  sigIntHandler.sa_flags = 0;
+  sigaction(SIGINT, &sigIntHandler, nullptr);
+}
+#endif
+/*!
+ * \brief GetCmdOption Parse and find the command option.
+ * \param argc arg counter
+ * \param argv arg values
+ * \param option command line option to search for.
+ * \param key whether the option itself is key
+ * \return value corresponding to option.
+ */
+string GetCmdOption(int argc, char* argv[], string option, bool key = false) {
+  string cmd;
+  for (int i = 1; i < argc; ++i) {
+    string arg = argv[i];
+    if (arg.find(option) == 0) {
+      if (key) {
+        cmd = argv[i];
+        return cmd;
+      }
+      // We assume "=" is the end of option.
+      ICHECK_EQ(*option.rbegin(), '=');
+      cmd = arg.substr(arg.find('=') + 1);
+      return cmd;
+    }
+  }
+  return cmd;
+}
+
+/*!
+ * \brief ParseCmdArgs parses the command line arguments.
+ * \param argc arg counter
+ * \param argv arg values
+ * \param args the output structure which holds the parsed values
+ */
+void ParseCmdArgs(int argc, char* argv[], struct ToolArgs& args) {
+  const string model = GetCmdOption(argc, argv, "--model=");
+  if (!model.empty()) {
+    args.model = model;
+  }
+
+  const string device = GetCmdOption(argc, argv, "--device=");
+  if (!device.empty()) {
+    args.device = device;
+  }
+
+  const string input = GetCmdOption(argc, argv, "--input=");
+  if (!input.empty()) {
+    args.input = input;
+  }
+
+  const string output = GetCmdOption(argc, argv, "--output=");
+  if (!output.empty()) {
+    args.output = output;
+  }
+
+  const string pmeta = GetCmdOption(argc, argv, "--dump-meta", true);
+  if (!pmeta.empty()) {
+    args.dump_meta = true;
+  }
+}
+
+/*!
+ * \brief Loads and Executes the model on given Target.
+ * \param args tool arguments
+ * \return result of operation.
+ */
+int ExecuteModel(ToolArgs& args) {
+#if defined(__linux__) || defined(__ANDROID__)
+  // Ctrl+C handler
+  HandleCtrlC();
+#endif
+
+  LOG(INFO) << "Welcome to executor";
+
+  // Initialize TVM Runner
+  TVMRunner runner = TVMRunner(args.model, args.device);
+
+  // Load the model
+  runner.Load();
+
+  // Query Model meta Information
+  TVMMetaInfo mInfo = runner.GetMetaInfo();
+
+  // Print Meta Information
+  if (args.dump_meta) runner.PrintMetaInfo();
+
+  if (args.input.empty() || args.output.empty()) {
+    LOG(INFO) << "Executing dry run ... ";
+    // Set random input for all inputs
+    for (auto& elem : mInfo.input_info) {
+      LOG(INFO) << "Set Random Input for :" << elem.first;
+      auto shape = elem.second.first;
+      size_t ssize = runner.GetInputMemSize(elem.first);
+      char* data = (char*)malloc(ssize);
+      // TODO: Ramdom initilalization

Review Comment:
   We could have a random initialization but doesn't matter for TVM as it can dry run with garbage data. Removing todo for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] echuraev commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

echuraev commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1073282477


##########
apps/cpp_rtvm/tvm_runner.cc:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm_runner.cc
+ * \brief TVM model runner implementation.
+ */
+
+#include "tvm_runner.h"
+
+#include <cnpy.h>
+
+#include <fstream>
+#include <streambuf>
+#include <string>
+
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief Get the TVM device id corresponding to device string.
+ * \param device the target device in string format.
+ * \return cl_device corresponding to the device string.
+ */
+int GetTVMDevice(std::string device) {
+  if (!device.compare("cl")) {
+    return static_cast<int>(kDLOpenCL);
+  } else {
+    LOG(FATAL) << "TVMRunner : Unsupported device :" << device;
+  }
+}
+
+/*!
+ * \brief Constructor for TVMRunner.
+ * \param path where the tfm compiler artifacts present.
+ * \param device the target device where we need to load the compiled model.
+ */
+TVMRunner::TVMRunner(std::string path, std::string device) : r_model_path(path), r_device(device) {
+  LOG(INFO) << "TVMRunner Constructor:" << r_model_path << " Devices:" << r_device;
+}
+
+/*!
+ * \brief Load Setup TVM graph runtime for given model.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::Load(void) {
+  LOG(INFO) << "TVMRunner Load:" << r_model_path;
+  // Load the lib file
+  r_mod_handle = Module::LoadFromFile((r_model_path + "/mod.so").c_str(), "so");
+
+  // Read model json file
+  std::ifstream json_reader((r_model_path + "/mod.json").c_str());
+  CHECK(!json_reader.fail()) << "Failed to open json file:" << (r_model_path + "/mod.json").c_str();
+  std::string json_str((std::istreambuf_iterator<char>(json_reader)),
+                       std::istreambuf_iterator<char>());
+  json_reader.close();
+
+  // Get ref to graph exeutor
+  auto f_handle = tvm::runtime::Registry::Get("tvm.graph_executor.create");
+
+  // Greate graph runtime
+  r_graph_handle = (*f_handle)(json_str, r_mod_handle, GetTVMDevice(r_device), 0);
+
+  // Read params binary file
+  std::ifstream params_reader((r_model_path + "/mod.params").c_str(), std::ios::binary);
+  CHECK(!params_reader.fail()) << "Failed to open json file:"
+                               << (r_model_path + "/mod.params").c_str();
+  const std::string params_str((std::istreambuf_iterator<char>(params_reader)),
+                               std::istreambuf_iterator<char>());
+  params_reader.close();
+  TVMByteArray params_arr;
+  params_arr.data = params_str.c_str();
+  params_arr.size = params_str.length();
+
+  // Load parameters
+  r_graph_handle.GetFunction("load_params")(params_arr);
+
+  return 0;
+}
+
+/*!
+ * \brief Calculated the memory size for the NDArray.
+ * \param NDArray object.
+ * \return size of the memory.
+ */
+inline size_t GetMemSize(NDArray& narr) {
+  size_t size = 1;
+  for (tvm_index_t i = 0; i < narr->ndim; ++i) {
+    size *= static_cast<size_t>(narr->shape[i]);
+  }
+  size *= (narr->dtype.bits * narr->dtype.lanes + 7) / 8;
+  return size;
+}
+
+/*!
+ * \brief Get the input alloc mem size.
+ * \param input_id The input id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetInputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetInputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Get the output alloc mem size.
+ * \param input_id The output id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetOutputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetOutputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_output")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Set the model inputs from npz file.
+ * \param inputfile the npz file from where we read input tensor data.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string inputfile) {
+  LOG(INFO) << "TVMRunner::SetInput (Numpy):" << inputfile;
+  cnpy::npz_t npz_input = cnpy::npz_load(inputfile);
+
+  for (auto& elem : mInfo.input_info) {
+    LOG(INFO) << "Set Numpy Input for :" << elem.first;
+    NDArray in_arr = r_graph_handle.GetFunction("get_input")(elem.first);
+    auto ssize = GetMemSize(in_arr);
+
+    if (npz_input.find(elem.first) != npz_input.end()) {
+      in_arr.CopyFromBytes(npz_input[elem.first].data<char>(), ssize);
+    } else {
+      LOG(WARNING) << "Couldn't find input " << elem.first << " in npy input file";
+    }
+  }
+
+  return 0;
+}
+
+/*!
+ * \brief Set the model input from the given binary buffer.
+ * \param input_id input node name.
+ * \param raw_input binary input buffer to copy over input NDArray.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string input_id, char* raw_input) {
+  LOG(INFO) << "TVMRunner::SetInput (Raw)";
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);

Review Comment:
   Ups... Sorry. My bad. Thank you!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] tqchen commented on pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

["tqchen (via GitHub)" <gi...@apache.org>]

tqchen commented on PR #13791:
URL: https://github.com/apache/tvm/pull/13791#issuecomment-1402008926

   One minor note: submodule inclusion would need to include license info, specifically,
   
   - add a section here https://github.com/apache/tvm/blob/main/LICENSE#L232
   - add license to https://github.com/apache/tvm/tree/main/licenses
   
   Please send a followup PR :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] srkreddy1238 commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

srkreddy1238 commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1071717825


##########
.gitmodules:
##########
@@ -19,3 +19,6 @@
 [submodule "3rdparty/OpenCL-Headers"]
 	path = 3rdparty/OpenCL-Headers
 	url = https://github.com/KhronosGroup/OpenCL-Headers.git
+[submodule "3rdparty/cnpy"]
+	path = 3rdparty/cnpy
+	url = https://github.com/rogersce/cnpy.git

Review Comment:
   I considered it initially to reuse what is available in TVM. But, this require additional effort from the developer/user to parse NDArray dump. Or we need to give a python utility that converts it to a numpy dump.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] echuraev merged pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

["echuraev (via GitHub)" <gi...@apache.org>]

echuraev merged PR #13791:
URL: https://github.com/apache/tvm/pull/13791


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] echuraev commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

echuraev commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1071746879


##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,220 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using ``rtvm` tool.

Review Comment:
   ```suggestion
   Overall process starts from getting a model from a framework all the way up to running on target device using `rtvm` tool.
   ```



##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,220 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using ``rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py can be used to download varius well known models from different frameworks.
+It will dump various models under model_data in current directory.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section RPC setup for the same.

Review Comment:
   ```suggestion
   Please refer below section [RPC setup](#rpc-setup) for the same.
   ```



##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,220 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using ``rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py can be used to download varius well known models from different frameworks.
+It will dump various models under model_data in current directory.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section RPC setup for the same.
+
+Auto tunng is necessary to obtain best performaning kernels. We can skip this step if we have tuning log already or the tuning cashe is available from tophub (inplicite by TVM compilation process).
+Below message indicate that there exists some kernels not optimized for the selected target. In this case we can proceed with tuning to best performance.
+```One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.```
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+the model to be tuned being ```./model_data/keras-resnet50/resnet50.h5```
+```
+
+the below command we can generate the tuning cache to file ```./model_data/keras-resnet50/keras-resnet50.log```
+
+```bash
+python3 -m tvm.driver.tvmc tune --target="opencl" --target-host="llvm -mtriple=aarch64-linux-gnu" \
+./model_data/keras-resnet50/resnet50.h5 -o ./model_data/keras-resnet50/keras-resnet50.log \
+--early-stopping 0 --repeat 30 --rpc-key android --rpc-tracker 127.0.0.1:9120 --trials 1024 \
+--tuning-records ./model_data/keras-resnet50/keras-resnet50-records.log --tuner xgb
+```
+
+where
+```bash
+--target="opencl -device=adreno" refers to opencl device on Android device
+--target-host="llvm -mtriple=aarch64-linux-gnu" refers to target_host being an ARM64 CPU
+Options --early-stopping, --repeat, --trials, --tuner are Auto TVM specific options. Please refer to AutoTVM documentation for more details here.
+```
+
+### Compile the model
+
+Compilation step generates TVM compiler output artifacts which need to be taken to target device for deployment.
+These artifacts is a compressed archive with kernel shared lib, json with cgaph description and params binary.
+
+Below command will generate the same
+
+
+```bash
+python3 -m tvm.driver.tvmc compile --cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang \
+--target="opencl, llvm" --target-llvm-mtriple aarch64-linux-gnu -o keras-resnet50.tar ./model_data/keras-resnet50/resnet50.h5
+```
+
+where
+```
+--cross-compiler : Indicates the cross compiler path for kernel library generation
+--target="opencl, llvm" indicates target and host devices
+--
+```
+
+### Test Run via RPC
+
+At this stage we can verify the generated compiler output for execution correctness over the RPC setup interface.
+Below command can run the compiled output on remote target device.
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+compilation out being keras-resnet50.tar
+```
+
+```bash
+python3 -m tvm.driver.tvmc run --device="cl" keras-resnet50.tar --rpc-key android --rpc-tracker 127.0.0.1:9120 --print-time
+```
+
+This inputs random inputs and validates the execution correctness of the compiled model.
+
+```tvmc``` tool has various options to input custom data, profile, benchmark the execution.
+
+
+### Deploy
+
+The tar archive generated can be used with ```rtvm``` application in Android native to run the same using tvm_runtime.
+
+
+# RPC Setup
+
+for Android devices require cross compilation of tvm_rpc (also libtvm_runtime.so which is a dependency) for remote device.
+RPC setup involved running tracker on host device and running tvm_rpc on target device.
+
+### Tracker
+
+below command runs the tracker on host over port ```9100```
+
+```bash
+python3 -m tvm.exec.rpc_tracker --host 127.0.0.1 --port 9100"
+```
+### RPC on Target
+
+With ```abcd1234ef``` being adb device id and tvm_rpc (and libtvm_runtime.so) is pushed to target device at ```/data/local/tmp/tvm_rpc/```
+
+```bash
+export ANDROID_SERIAL=abcd1234ef
+# Below settings will reroute networking tcm connections on devices to host device via adb interface
+adb reverse tcp:9100 tcp:9100
+adb forward tcp:5000 tcp:5000
+# Run the tvm_rpc on device
+env adb shell "cd /data/local/tmp/tvm_rpc; killall -9 tvm_rpc; \
+LD_LIBRARY_PATH=/data/local/tmp/tvm_rpc/ ./tvm_rpc server --host=0.0.0.0 --port=5000 --port-end=5010 --tracker=127.0.0.1:9100 --key=android
+```
+
+Now we have the rpc setup with ```--rpc-tracker=27.0.0.1:9100``` and ```--rpc-key=android```.
+

Review Comment:
   Add information how the user can check that the device has successfully connected to tracker. I mean: `python3 -m tvm.exec.query_rpc_tracker --port 9100`



##########
apps/cpp_rtvm/tvm_runner.cc:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm_runner.cc
+ * \brief TVM model runner implementation.
+ */
+
+#include "tvm_runner.h"
+
+#include <cnpy.h>
+
+#include <fstream>
+#include <streambuf>
+#include <string>
+
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief Get the TVM device id corresponding to device string.
+ * \param device the target device in string format.
+ * \return cl_device corresponding to the device string.
+ */
+int GetTVMDevice(std::string device) {
+  if (!device.compare("cl")) {
+    return static_cast<int>(kDLOpenCL);

Review Comment:
   For now `rtvm` can be used only for running model on OpenCL device. Should it be extended to other devices (e.g. cpu or others)?



##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,220 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using ``rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py can be used to download varius well known models from different frameworks.
+It will dump various models under model_data in current directory.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section RPC setup for the same.
+
+Auto tunng is necessary to obtain best performaning kernels. We can skip this step if we have tuning log already or the tuning cashe is available from tophub (inplicite by TVM compilation process).
+Below message indicate that there exists some kernels not optimized for the selected target. In this case we can proceed with tuning to best performance.
+```One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.```
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+the model to be tuned being ```./model_data/keras-resnet50/resnet50.h5```
+```
+
+the below command we can generate the tuning cache to file ```./model_data/keras-resnet50/keras-resnet50.log```
+
+```bash
+python3 -m tvm.driver.tvmc tune --target="opencl" --target-host="llvm -mtriple=aarch64-linux-gnu" \
+./model_data/keras-resnet50/resnet50.h5 -o ./model_data/keras-resnet50/keras-resnet50.log \
+--early-stopping 0 --repeat 30 --rpc-key android --rpc-tracker 127.0.0.1:9120 --trials 1024 \
+--tuning-records ./model_data/keras-resnet50/keras-resnet50-records.log --tuner xgb
+```
+
+where
+```bash
+--target="opencl -device=adreno" refers to opencl device on Android device
+--target-host="llvm -mtriple=aarch64-linux-gnu" refers to target_host being an ARM64 CPU
+Options --early-stopping, --repeat, --trials, --tuner are Auto TVM specific options. Please refer to AutoTVM documentation for more details here.
+```
+
+### Compile the model
+
+Compilation step generates TVM compiler output artifacts which need to be taken to target device for deployment.
+These artifacts is a compressed archive with kernel shared lib, json with cgaph description and params binary.
+
+Below command will generate the same
+
+
+```bash
+python3 -m tvm.driver.tvmc compile --cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang \
+--target="opencl, llvm" --target-llvm-mtriple aarch64-linux-gnu -o keras-resnet50.tar ./model_data/keras-resnet50/resnet50.h5
+```
+
+where
+```
+--cross-compiler : Indicates the cross compiler path for kernel library generation
+--target="opencl, llvm" indicates target and host devices
+--
+```
+
+### Test Run via RPC
+
+At this stage we can verify the generated compiler output for execution correctness over the RPC setup interface.
+Below command can run the compiled output on remote target device.
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```
+compilation out being keras-resnet50.tar
+```
+
+```bash
+python3 -m tvm.driver.tvmc run --device="cl" keras-resnet50.tar --rpc-key android --rpc-tracker 127.0.0.1:9120 --print-time
+```
+
+This inputs random inputs and validates the execution correctness of the compiled model.
+
+```tvmc``` tool has various options to input custom data, profile, benchmark the execution.
+
+
+### Deploy
+
+The tar archive generated can be used with ```rtvm``` application in Android native to run the same using tvm_runtime.
+
+
+# RPC Setup
+
+for Android devices require cross compilation of tvm_rpc (also libtvm_runtime.so which is a dependency) for remote device.
+RPC setup involved running tracker on host device and running tvm_rpc on target device.
+
+### Tracker
+
+below command runs the tracker on host over port ```9100```
+
+```bash
+python3 -m tvm.exec.rpc_tracker --host 127.0.0.1 --port 9100"
+```
+### RPC on Target
+
+With ```abcd1234ef``` being adb device id and tvm_rpc (and libtvm_runtime.so) is pushed to target device at ```/data/local/tmp/tvm_rpc/```
+
+```bash
+export ANDROID_SERIAL=abcd1234ef
+# Below settings will reroute networking tcm connections on devices to host device via adb interface
+adb reverse tcp:9100 tcp:9100
+adb forward tcp:5000 tcp:5000
+# Run the tvm_rpc on device
+env adb shell "cd /data/local/tmp/tvm_rpc; killall -9 tvm_rpc; \
+LD_LIBRARY_PATH=/data/local/tmp/tvm_rpc/ ./tvm_rpc server --host=0.0.0.0 --port=5000 --port-end=5010 --tracker=127.0.0.1:9100 --key=android
+```
+
+Now we have the rpc setup with ```--rpc-tracker=27.0.0.1:9100``` and ```--rpc-key=android```.
+
+
+# Target Specific Configuration
+
+Below sections describe device/target specific settings to be used with tvmc tool
+
+### Adreno GPU
+
+Adreno GPU has a docker defined that helps to ease the development environment.
+
+Below command builds host and target rpc components for Adreno and drops into an interactive shell.
+
+```bash
+./tests/scripts/ci.py adreno -i
+```
+
+Also, one can build with Adreno OpenCLML SDK support
+
+```bash
+export ADRENO_OPENCL=<Path to OpenCLML SDK>
+./tests/scripts/ci.py adreno -i
+```
+
+Above command produces
+```build-adreno``` which is host build
+```build-adreno-target``` which contains cross compiled tvm_rpc and libtvm_runtime.so
+
+
+Below options to be used for Adreno GPU while working with tvmc
+
+* Tuning
+
+```
+--target="opencl -device=adreno"
+--target-host="llvm -mtriple=aarch64-linux-gnu"
+```
+
+* Compilation
+
+```
+--cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang
+--target="opencl, llvm"
+--target-opencl-device adreno
+--target-llvm-mtriple aarch64-linux-gnu
+```
+
+While enabling CLML just need to specify below target option for compilation.
+```--target="opencl, clml, llvm"```
+
+
+* Running
+
+```--device="cl"```
+
+
+For example with a model from keras ```./model_data/keras-resnet50/resnet50.h5```
+
+
+```bash
+# Tuning
+python3 -m tvm.driver.tvmc tune --desired-layout NCHW --target="opencl -device=adreno" --target-host="llvm -mtriple=aarch64-linux-gnu" \
+./model_data/keras-resnet50/resnet50.h5 -o ./model_data/keras-resnet50/keras-resnet50.log --early-stopping 0 --repeat 30 \
+--rpc-key android --rpc-tracker 127.0.0.1:9120 --trials 1024 --tuning-records ./model_data/keras-resnet50/keras-resnet50-records.log --tuner xgb
+
+# Tuning produces tuning log ./model_data/keras-resnet50/keras-resnet50.log
+
+
+# Compilation
+python3 -m tvm.driver.tvmc compile --cross-compiler ${ANDROID_NDK_HOME}/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang \
+--desired-layout NCHW --target="opencl, llvm" --target-opencl-device adreno --target-llvm-mtriple aarch64-linux-gnu \
+./model_data/keras-resnet50/resnet50.h5 -o keras-resnet50.tar
+
+# Compilation produces target artifacts keras-resnet50.tar
+
+# Run on adreno device via RPC
+# Assuming tracker is running on 127.0.0.1:9190 and target key is "android"
+python3 -m tvm.driver.tvmc run --device="cl" keras-resnet50.tar --rpc-key android --rpc-tracker 127.0.0.1:9120 --print-time
+
+```

Review Comment:
   Sorry, probably I missed something. But in this tutorial, you only use `tvmc` and didn't show how to use `rtvm` and what are the benefits of using `rtvm`.



##########
apps/cpp_rtvm/tvm_runner.cc:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm_runner.cc
+ * \brief TVM model runner implementation.
+ */
+
+#include "tvm_runner.h"
+
+#include <cnpy.h>
+
+#include <fstream>
+#include <streambuf>
+#include <string>
+
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief Get the TVM device id corresponding to device string.
+ * \param device the target device in string format.
+ * \return cl_device corresponding to the device string.
+ */
+int GetTVMDevice(std::string device) {
+  if (!device.compare("cl")) {
+    return static_cast<int>(kDLOpenCL);
+  } else {
+    LOG(FATAL) << "TVMRunner : Unsupported device :" << device;
+  }
+}
+
+/*!
+ * \brief Constructor for TVMRunner.
+ * \param path where the tfm compiler artifacts present.
+ * \param device the target device where we need to load the compiled model.
+ */
+TVMRunner::TVMRunner(std::string path, std::string device) : r_model_path(path), r_device(device) {
+  LOG(INFO) << "TVMRunner Constructor:" << r_model_path << " Devices:" << r_device;
+}
+
+/*!
+ * \brief Load Setup TVM graph runtime for given model.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::Load(void) {
+  LOG(INFO) << "TVMRunner Load:" << r_model_path;
+  // Load the lib file
+  r_mod_handle = Module::LoadFromFile((r_model_path + "/mod.so").c_str(), "so");
+
+  // Read model json file
+  std::ifstream json_reader((r_model_path + "/mod.json").c_str());
+  CHECK(!json_reader.fail()) << "Failed to open json file:" << (r_model_path + "/mod.json").c_str();
+  std::string json_str((std::istreambuf_iterator<char>(json_reader)),
+                       std::istreambuf_iterator<char>());
+  json_reader.close();
+
+  // Get ref to graph exeutor
+  auto f_handle = tvm::runtime::Registry::Get("tvm.graph_executor.create");
+
+  // Greate graph runtime
+  r_graph_handle = (*f_handle)(json_str, r_mod_handle, GetTVMDevice(r_device), 0);
+
+  // Read params binary file
+  std::ifstream params_reader((r_model_path + "/mod.params").c_str(), std::ios::binary);
+  CHECK(!params_reader.fail()) << "Failed to open json file:"
+                               << (r_model_path + "/mod.params").c_str();
+  const std::string params_str((std::istreambuf_iterator<char>(params_reader)),
+                               std::istreambuf_iterator<char>());
+  params_reader.close();
+  TVMByteArray params_arr;
+  params_arr.data = params_str.c_str();
+  params_arr.size = params_str.length();
+
+  // Load parameters
+  r_graph_handle.GetFunction("load_params")(params_arr);
+
+  return 0;
+}
+
+/*!
+ * \brief Calculated the memory size for the NDArray.
+ * \param NDArray object.
+ * \return size of the memory.
+ */
+inline size_t GetMemSize(NDArray& narr) {
+  size_t size = 1;
+  for (tvm_index_t i = 0; i < narr->ndim; ++i) {
+    size *= static_cast<size_t>(narr->shape[i]);
+  }
+  size *= (narr->dtype.bits * narr->dtype.lanes + 7) / 8;
+  return size;
+}
+
+/*!
+ * \brief Get the input alloc mem size.
+ * \param input_id The input id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetInputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetInputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Get the output alloc mem size.
+ * \param input_id The output id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetOutputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetOutputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_output")(input_id);
+  auto ssize = GetMemSize(in_arr);

Review Comment:
   ```suggestion
   size_t TVMRunner::GetOutputMemSize(std::string output_id) {
     LOG(INFO) << "TVMRunner::GetOutputMemSize:" << output_id;
   
     NDArray out_arr = r_graph_handle.GetFunction("get_output")(output_id);
     auto ssize = GetMemSize(out_arr);
   ```



##########
apps/cpp_rtvm/README.md:
##########
@@ -0,0 +1,220 @@
+<!--- Licensed to the Apache Software Foundation (ASF) under one -->
+<!--- or more contributor license agreements.  See the NOTICE file -->
+<!--- distributed with this work for additional information -->
+<!--- regarding copyright ownership.  The ASF licenses this file -->
+<!--- to you under the Apache License, Version 2.0 (the -->
+<!--- "License"); you may not use this file except in compliance -->
+<!--- with the License.  You may obtain a copy of the License at -->
+
+<!---   http://www.apache.org/licenses/LICENSE-2.0 -->
+
+<!--- Unless required by applicable law or agreed to in writing, -->
+<!--- software distributed under the License is distributed on an -->
+<!--- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -->
+<!--- KIND, either express or implied.  See the License for the -->
+<!--- specific language governing permissions and limitations -->
+<!--- under the License. -->
+
+
+# Native Inference application for CPP Native
+
+Native inference tool ```rtvm``` helps in deploying TVM compiled models from a standalone cpp environment.
+Overall process starts from getting a model from a framework all the way up to running on target device using ``rtvm` tool.
+
+### Models
+
+Models can be downloaded from well known frameworks like Tensorflow, PyTorch, TFLite, Onnx ..etc.
+scripts/download_models.py can be used to download varius well known models from different frameworks.
+It will dump various models under model_data in current directory.
+
+```bash
+python3  scripts/download_models.py
+```
+
+### Auto Tuning
+Auto tuning process tunes various operatrors the given model for respective target. Auto tuning for remote devices use ```tvm_rpc``` and we need to setup the rpc environment before we invoke tuning.
+Please refer below section RPC setup for the same.
+
+Auto tunng is necessary to obtain best performaning kernels. We can skip this step if we have tuning log already or the tuning cashe is available from tophub (inplicite by TVM compilation process).
+Below message indicate that there exists some kernels not optimized for the selected target. In this case we can proceed with tuning to best performance.
+```One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.```
+
+with
+
+``` bash
+tvm tracker running on ```TVM_TRACKER_HOST```
+tracker port being ```TVM_TRACKER_PORT```
+rpc device access key being ```TVM_RPC_KEY```

Review Comment:
   If you are new in TVM and read this documentation first time then it might be not obviously that `TVM_TRACKER_HOST`, `TVM_TRACKER_PORT` and `TVM_RPC_KEY` are environment variables which should be initialized by the user. Could you please add a bit more details about the configuration process?



##########
apps/cpp_rtvm/scripts/download_models.py:
##########
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+tmp_dir = "./model_data/"
+dload_models = []
+
+# Keras : Resnet50

Review Comment:
   In the `README.md` was written that this script can be used to `download varius well known models from different frameworks`. Now it looks like this script can download only ResNet50 from Keras. Probably the `README.md` should be updated, and we should mention that this script can be used as a template, and it should be extended by the user to download other models? 



##########
apps/cpp_rtvm/main.cc:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file main.cc
+ * \brief TVM runtime utility for TVM.
+ */
+#include <csignal>
+#include <cstdio>
+#include <cstdlib>
+#if defined(__linux__) || defined(__ANDROID__)
+#include <unistd.h>
+#endif
+#include <dmlc/logging.h>
+
+#include <cstring>
+#include <iostream>
+#include <sstream>
+#include <vector>
+
+#include "../../src/support/socket.h"
+#include "../../src/support/utils.h"
+#include "tvm_runner.h"
+
+#if defined(_WIN32)
+#include "win32_process.h"
+#endif
+
+using namespace std;
+using namespace tvm::runtime;
+using namespace tvm::support;
+
+static const string kUsage =
+    "Command line usage\n"
+    "--model        - The tvm artifact to load\n"
+    "--device       - The target device to use {llvm, cl, ...etc.}\n"
+    "--input        - Numpy file for the model input (optional and we use random of not given)\n"
+    "--output       - Numpy file name to dump the model output as numpy\n"
+    "--dump-meta    - Dump model meta information\n"
+    "\n"
+    "  Example\n"
+    "  ./rtvm --model keras-resnet50 --device=\"cl --dump-mata\"\n"
+    "  ./rtvm --model keras-resnet50 --device=\"cl\" --input input.npz --output=output.npz\n"
+    "\n";
+
+/*!
+ * \brief Tool Arguments.
+ * \arg model The tvm artifact to load & run
+ * \arg device The target device to use {llvm, cl, ...etc.}
+ * \arg input Numpy file for the model input
+ * \arg output Numpy file name to dump the model output as numpy
+ */
+struct ToolArgs {
+  string model;
+  string device;
+  string input;
+  string output;
+  bool dump_meta = false;
+};
+
+/*!
+ * \brief PrintArgs print the contents of ToolArgs
+ * \param args ToolArgs structure
+ */
+void PrintArgs(const ToolArgs& args) {
+  LOG(INFO) << "Model         = " << args.model;
+  LOG(INFO) << "Device        = " << args.device;
+  LOG(INFO) << "Input         = " << args.input;
+  LOG(INFO) << "Output        = " << args.output;
+  LOG(INFO) << "Dump Metadata = " << ((args.dump_meta) ? ("True") : ("False"));
+}
+
+#if defined(__linux__) || defined(__ANDROID__)
+/*!
+ * \brief CtrlCHandler, exits if Ctrl+C is pressed
+ * \param s signal
+ */
+void CtrlCHandler(int s) {
+  LOG(INFO) << "\nUser pressed Ctrl+C, Exiting";
+  exit(1);
+}
+
+/*!
+ * \brief HandleCtrlC Register for handling Ctrl+C event.
+ */
+void HandleCtrlC() {
+  // Ctrl+C handler
+  struct sigaction sigIntHandler;
+  sigIntHandler.sa_handler = CtrlCHandler;
+  sigemptyset(&sigIntHandler.sa_mask);
+  sigIntHandler.sa_flags = 0;
+  sigaction(SIGINT, &sigIntHandler, nullptr);
+}
+#endif
+/*!
+ * \brief GetCmdOption Parse and find the command option.
+ * \param argc arg counter
+ * \param argv arg values
+ * \param option command line option to search for.
+ * \param key whether the option itself is key
+ * \return value corresponding to option.
+ */
+string GetCmdOption(int argc, char* argv[], string option, bool key = false) {
+  string cmd;
+  for (int i = 1; i < argc; ++i) {
+    string arg = argv[i];
+    if (arg.find(option) == 0) {
+      if (key) {
+        cmd = argv[i];
+        return cmd;
+      }
+      // We assume "=" is the end of option.
+      ICHECK_EQ(*option.rbegin(), '=');
+      cmd = arg.substr(arg.find('=') + 1);
+      return cmd;
+    }
+  }
+  return cmd;
+}
+
+/*!
+ * \brief ParseCmdArgs parses the command line arguments.
+ * \param argc arg counter
+ * \param argv arg values
+ * \param args the output structure which holds the parsed values
+ */
+void ParseCmdArgs(int argc, char* argv[], struct ToolArgs& args) {
+  const string model = GetCmdOption(argc, argv, "--model=");
+  if (!model.empty()) {
+    args.model = model;
+  }
+
+  const string device = GetCmdOption(argc, argv, "--device=");
+  if (!device.empty()) {
+    args.device = device;
+  }
+
+  const string input = GetCmdOption(argc, argv, "--input=");
+  if (!input.empty()) {
+    args.input = input;
+  }
+
+  const string output = GetCmdOption(argc, argv, "--output=");
+  if (!output.empty()) {
+    args.output = output;
+  }
+
+  const string pmeta = GetCmdOption(argc, argv, "--dump-meta", true);
+  if (!pmeta.empty()) {
+    args.dump_meta = true;
+  }
+}
+
+/*!
+ * \brief Loads and Executes the model on given Target.
+ * \param args tool arguments
+ * \return result of operation.
+ */
+int ExecuteModel(ToolArgs& args) {
+#if defined(__linux__) || defined(__ANDROID__)
+  // Ctrl+C handler
+  HandleCtrlC();
+#endif
+
+  LOG(INFO) << "Welcome to executor";
+
+  // Initialize TVM Runner
+  TVMRunner runner = TVMRunner(args.model, args.device);
+
+  // Load the model
+  runner.Load();
+
+  // Query Model meta Information
+  TVMMetaInfo mInfo = runner.GetMetaInfo();
+
+  // Print Meta Information
+  if (args.dump_meta) runner.PrintMetaInfo();
+
+  if (args.input.empty() || args.output.empty()) {
+    LOG(INFO) << "Executing dry run ... ";
+    // Set random input for all inputs
+    for (auto& elem : mInfo.input_info) {
+      LOG(INFO) << "Set Random Input for :" << elem.first;
+      auto shape = elem.second.first;
+      size_t ssize = runner.GetInputMemSize(elem.first);
+      char* data = (char*)malloc(ssize);
+      // TODO: Ramdom initilalization

Review Comment:
   When this TODO comment should be resolved?



##########
apps/cpp_rtvm/main.cc:
##########
@@ -0,0 +1,260 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file main.cc
+ * \brief TVM runtime utility for TVM.
+ */
+#include <csignal>
+#include <cstdio>
+#include <cstdlib>
+#if defined(__linux__) || defined(__ANDROID__)
+#include <unistd.h>
+#endif
+#include <dmlc/logging.h>
+
+#include <cstring>
+#include <iostream>
+#include <sstream>
+#include <vector>
+
+#include "../../src/support/socket.h"
+#include "../../src/support/utils.h"
+#include "tvm_runner.h"
+
+#if defined(_WIN32)
+#include "win32_process.h"
+#endif
+
+using namespace std;
+using namespace tvm::runtime;
+using namespace tvm::support;
+
+static const string kUsage =
+    "Command line usage\n"
+    "--model        - The tvm artifact to load\n"
+    "--device       - The target device to use {llvm, cl, ...etc.}\n"
+    "--input        - Numpy file for the model input (optional and we use random of not given)\n"
+    "--output       - Numpy file name to dump the model output as numpy\n"
+    "--dump-meta    - Dump model meta information\n"
+    "\n"
+    "  Example\n"
+    "  ./rtvm --model keras-resnet50 --device=\"cl --dump-mata\"\n"
+    "  ./rtvm --model keras-resnet50 --device=\"cl\" --input input.npz --output=output.npz\n"
+    "\n";
+
+/*!
+ * \brief Tool Arguments.
+ * \arg model The tvm artifact to load & run
+ * \arg device The target device to use {llvm, cl, ...etc.}
+ * \arg input Numpy file for the model input
+ * \arg output Numpy file name to dump the model output as numpy
+ */
+struct ToolArgs {
+  string model;
+  string device;
+  string input;
+  string output;
+  bool dump_meta = false;
+};
+
+/*!
+ * \brief PrintArgs print the contents of ToolArgs
+ * \param args ToolArgs structure
+ */
+void PrintArgs(const ToolArgs& args) {
+  LOG(INFO) << "Model         = " << args.model;
+  LOG(INFO) << "Device        = " << args.device;
+  LOG(INFO) << "Input         = " << args.input;
+  LOG(INFO) << "Output        = " << args.output;
+  LOG(INFO) << "Dump Metadata = " << ((args.dump_meta) ? ("True") : ("False"));
+}
+
+#if defined(__linux__) || defined(__ANDROID__)
+/*!
+ * \brief CtrlCHandler, exits if Ctrl+C is pressed
+ * \param s signal
+ */
+void CtrlCHandler(int s) {
+  LOG(INFO) << "\nUser pressed Ctrl+C, Exiting";
+  exit(1);
+}
+
+/*!
+ * \brief HandleCtrlC Register for handling Ctrl+C event.
+ */
+void HandleCtrlC() {
+  // Ctrl+C handler
+  struct sigaction sigIntHandler;
+  sigIntHandler.sa_handler = CtrlCHandler;
+  sigemptyset(&sigIntHandler.sa_mask);
+  sigIntHandler.sa_flags = 0;
+  sigaction(SIGINT, &sigIntHandler, nullptr);
+}
+#endif
+/*!
+ * \brief GetCmdOption Parse and find the command option.
+ * \param argc arg counter
+ * \param argv arg values
+ * \param option command line option to search for.
+ * \param key whether the option itself is key
+ * \return value corresponding to option.
+ */
+string GetCmdOption(int argc, char* argv[], string option, bool key = false) {
+  string cmd;
+  for (int i = 1; i < argc; ++i) {
+    string arg = argv[i];
+    if (arg.find(option) == 0) {
+      if (key) {
+        cmd = argv[i];
+        return cmd;
+      }
+      // We assume "=" is the end of option.
+      ICHECK_EQ(*option.rbegin(), '=');
+      cmd = arg.substr(arg.find('=') + 1);
+      return cmd;
+    }
+  }
+  return cmd;
+}
+
+/*!
+ * \brief ParseCmdArgs parses the command line arguments.
+ * \param argc arg counter
+ * \param argv arg values
+ * \param args the output structure which holds the parsed values
+ */
+void ParseCmdArgs(int argc, char* argv[], struct ToolArgs& args) {
+  const string model = GetCmdOption(argc, argv, "--model=");
+  if (!model.empty()) {
+    args.model = model;

Review Comment:
   What if the `model` is empty? In this case, you won't update `args.model` and which model should be executed by TVM?



##########
apps/cpp_rtvm/tvm_runner.cc:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm_runner.cc
+ * \brief TVM model runner implementation.
+ */
+
+#include "tvm_runner.h"
+
+#include <cnpy.h>
+
+#include <fstream>
+#include <streambuf>
+#include <string>
+
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief Get the TVM device id corresponding to device string.
+ * \param device the target device in string format.
+ * \return cl_device corresponding to the device string.
+ */
+int GetTVMDevice(std::string device) {
+  if (!device.compare("cl")) {
+    return static_cast<int>(kDLOpenCL);
+  } else {
+    LOG(FATAL) << "TVMRunner : Unsupported device :" << device;
+  }
+}
+
+/*!
+ * \brief Constructor for TVMRunner.
+ * \param path where the tfm compiler artifacts present.
+ * \param device the target device where we need to load the compiled model.
+ */
+TVMRunner::TVMRunner(std::string path, std::string device) : r_model_path(path), r_device(device) {
+  LOG(INFO) << "TVMRunner Constructor:" << r_model_path << " Devices:" << r_device;
+}
+
+/*!
+ * \brief Load Setup TVM graph runtime for given model.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::Load(void) {
+  LOG(INFO) << "TVMRunner Load:" << r_model_path;
+  // Load the lib file
+  r_mod_handle = Module::LoadFromFile((r_model_path + "/mod.so").c_str(), "so");
+
+  // Read model json file
+  std::ifstream json_reader((r_model_path + "/mod.json").c_str());
+  CHECK(!json_reader.fail()) << "Failed to open json file:" << (r_model_path + "/mod.json").c_str();
+  std::string json_str((std::istreambuf_iterator<char>(json_reader)),
+                       std::istreambuf_iterator<char>());
+  json_reader.close();
+
+  // Get ref to graph exeutor
+  auto f_handle = tvm::runtime::Registry::Get("tvm.graph_executor.create");
+
+  // Greate graph runtime
+  r_graph_handle = (*f_handle)(json_str, r_mod_handle, GetTVMDevice(r_device), 0);
+
+  // Read params binary file
+  std::ifstream params_reader((r_model_path + "/mod.params").c_str(), std::ios::binary);
+  CHECK(!params_reader.fail()) << "Failed to open json file:"
+                               << (r_model_path + "/mod.params").c_str();
+  const std::string params_str((std::istreambuf_iterator<char>(params_reader)),
+                               std::istreambuf_iterator<char>());
+  params_reader.close();
+  TVMByteArray params_arr;
+  params_arr.data = params_str.c_str();
+  params_arr.size = params_str.length();
+
+  // Load parameters
+  r_graph_handle.GetFunction("load_params")(params_arr);
+
+  return 0;
+}
+
+/*!
+ * \brief Calculated the memory size for the NDArray.
+ * \param NDArray object.
+ * \return size of the memory.
+ */
+inline size_t GetMemSize(NDArray& narr) {
+  size_t size = 1;
+  for (tvm_index_t i = 0; i < narr->ndim; ++i) {
+    size *= static_cast<size_t>(narr->shape[i]);
+  }
+  size *= (narr->dtype.bits * narr->dtype.lanes + 7) / 8;
+  return size;
+}
+
+/*!
+ * \brief Get the input alloc mem size.
+ * \param input_id The input id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetInputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetInputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Get the output alloc mem size.
+ * \param input_id The output id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetOutputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetOutputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_output")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Set the model inputs from npz file.
+ * \param inputfile the npz file from where we read input tensor data.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string inputfile) {
+  LOG(INFO) << "TVMRunner::SetInput (Numpy):" << inputfile;
+  cnpy::npz_t npz_input = cnpy::npz_load(inputfile);
+
+  for (auto& elem : mInfo.input_info) {
+    LOG(INFO) << "Set Numpy Input for :" << elem.first;
+    NDArray in_arr = r_graph_handle.GetFunction("get_input")(elem.first);
+    auto ssize = GetMemSize(in_arr);

Review Comment:
   It is the same, isn't it?
   ```suggestion
       auto ssize = GetInputMemSize(elem.first);
   ```



##########
apps/cpp_rtvm/tvm_runner.cc:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm_runner.cc
+ * \brief TVM model runner implementation.
+ */
+
+#include "tvm_runner.h"
+
+#include <cnpy.h>
+
+#include <fstream>
+#include <streambuf>
+#include <string>
+
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief Get the TVM device id corresponding to device string.
+ * \param device the target device in string format.
+ * \return cl_device corresponding to the device string.
+ */
+int GetTVMDevice(std::string device) {
+  if (!device.compare("cl")) {
+    return static_cast<int>(kDLOpenCL);
+  } else {
+    LOG(FATAL) << "TVMRunner : Unsupported device :" << device;
+  }
+}
+
+/*!
+ * \brief Constructor for TVMRunner.
+ * \param path where the tfm compiler artifacts present.
+ * \param device the target device where we need to load the compiled model.
+ */
+TVMRunner::TVMRunner(std::string path, std::string device) : r_model_path(path), r_device(device) {
+  LOG(INFO) << "TVMRunner Constructor:" << r_model_path << " Devices:" << r_device;
+}
+
+/*!
+ * \brief Load Setup TVM graph runtime for given model.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::Load(void) {
+  LOG(INFO) << "TVMRunner Load:" << r_model_path;
+  // Load the lib file
+  r_mod_handle = Module::LoadFromFile((r_model_path + "/mod.so").c_str(), "so");
+
+  // Read model json file
+  std::ifstream json_reader((r_model_path + "/mod.json").c_str());
+  CHECK(!json_reader.fail()) << "Failed to open json file:" << (r_model_path + "/mod.json").c_str();
+  std::string json_str((std::istreambuf_iterator<char>(json_reader)),
+                       std::istreambuf_iterator<char>());
+  json_reader.close();
+
+  // Get ref to graph exeutor
+  auto f_handle = tvm::runtime::Registry::Get("tvm.graph_executor.create");
+
+  // Greate graph runtime
+  r_graph_handle = (*f_handle)(json_str, r_mod_handle, GetTVMDevice(r_device), 0);
+
+  // Read params binary file
+  std::ifstream params_reader((r_model_path + "/mod.params").c_str(), std::ios::binary);
+  CHECK(!params_reader.fail()) << "Failed to open json file:"
+                               << (r_model_path + "/mod.params").c_str();
+  const std::string params_str((std::istreambuf_iterator<char>(params_reader)),
+                               std::istreambuf_iterator<char>());
+  params_reader.close();
+  TVMByteArray params_arr;
+  params_arr.data = params_str.c_str();
+  params_arr.size = params_str.length();
+
+  // Load parameters
+  r_graph_handle.GetFunction("load_params")(params_arr);
+
+  return 0;
+}
+
+/*!
+ * \brief Calculated the memory size for the NDArray.
+ * \param NDArray object.
+ * \return size of the memory.
+ */
+inline size_t GetMemSize(NDArray& narr) {
+  size_t size = 1;
+  for (tvm_index_t i = 0; i < narr->ndim; ++i) {
+    size *= static_cast<size_t>(narr->shape[i]);
+  }
+  size *= (narr->dtype.bits * narr->dtype.lanes + 7) / 8;
+  return size;
+}
+
+/*!
+ * \brief Get the input alloc mem size.
+ * \param input_id The input id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetInputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetInputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Get the output alloc mem size.
+ * \param input_id The output id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetOutputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetOutputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_output")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Set the model inputs from npz file.
+ * \param inputfile the npz file from where we read input tensor data.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string inputfile) {
+  LOG(INFO) << "TVMRunner::SetInput (Numpy):" << inputfile;
+  cnpy::npz_t npz_input = cnpy::npz_load(inputfile);
+
+  for (auto& elem : mInfo.input_info) {
+    LOG(INFO) << "Set Numpy Input for :" << elem.first;
+    NDArray in_arr = r_graph_handle.GetFunction("get_input")(elem.first);
+    auto ssize = GetMemSize(in_arr);
+
+    if (npz_input.find(elem.first) != npz_input.end()) {
+      in_arr.CopyFromBytes(npz_input[elem.first].data<char>(), ssize);
+    } else {
+      LOG(WARNING) << "Couldn't find input " << elem.first << " in npy input file";
+    }
+  }
+
+  return 0;
+}
+
+/*!
+ * \brief Set the model input from the given binary buffer.
+ * \param input_id input node name.
+ * \param raw_input binary input buffer to copy over input NDArray.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string input_id, char* raw_input) {
+  LOG(INFO) << "TVMRunner::SetInput (Raw)";
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);

Review Comment:
   The same comments for `GetOutput` methods. 



##########
apps/cpp_rtvm/tvm_runner.cc:
##########
@@ -0,0 +1,304 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm_runner.cc
+ * \brief TVM model runner implementation.
+ */
+
+#include "tvm_runner.h"
+
+#include <cnpy.h>
+
+#include <fstream>
+#include <streambuf>
+#include <string>
+
+namespace tvm {
+namespace runtime {
+
+/*!
+ * \brief Get the TVM device id corresponding to device string.
+ * \param device the target device in string format.
+ * \return cl_device corresponding to the device string.
+ */
+int GetTVMDevice(std::string device) {
+  if (!device.compare("cl")) {
+    return static_cast<int>(kDLOpenCL);
+  } else {
+    LOG(FATAL) << "TVMRunner : Unsupported device :" << device;
+  }
+}
+
+/*!
+ * \brief Constructor for TVMRunner.
+ * \param path where the tfm compiler artifacts present.
+ * \param device the target device where we need to load the compiled model.
+ */
+TVMRunner::TVMRunner(std::string path, std::string device) : r_model_path(path), r_device(device) {
+  LOG(INFO) << "TVMRunner Constructor:" << r_model_path << " Devices:" << r_device;
+}
+
+/*!
+ * \brief Load Setup TVM graph runtime for given model.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::Load(void) {
+  LOG(INFO) << "TVMRunner Load:" << r_model_path;
+  // Load the lib file
+  r_mod_handle = Module::LoadFromFile((r_model_path + "/mod.so").c_str(), "so");
+
+  // Read model json file
+  std::ifstream json_reader((r_model_path + "/mod.json").c_str());
+  CHECK(!json_reader.fail()) << "Failed to open json file:" << (r_model_path + "/mod.json").c_str();
+  std::string json_str((std::istreambuf_iterator<char>(json_reader)),
+                       std::istreambuf_iterator<char>());
+  json_reader.close();
+
+  // Get ref to graph exeutor
+  auto f_handle = tvm::runtime::Registry::Get("tvm.graph_executor.create");
+
+  // Greate graph runtime
+  r_graph_handle = (*f_handle)(json_str, r_mod_handle, GetTVMDevice(r_device), 0);
+
+  // Read params binary file
+  std::ifstream params_reader((r_model_path + "/mod.params").c_str(), std::ios::binary);
+  CHECK(!params_reader.fail()) << "Failed to open json file:"
+                               << (r_model_path + "/mod.params").c_str();
+  const std::string params_str((std::istreambuf_iterator<char>(params_reader)),
+                               std::istreambuf_iterator<char>());
+  params_reader.close();
+  TVMByteArray params_arr;
+  params_arr.data = params_str.c_str();
+  params_arr.size = params_str.length();
+
+  // Load parameters
+  r_graph_handle.GetFunction("load_params")(params_arr);
+
+  return 0;
+}
+
+/*!
+ * \brief Calculated the memory size for the NDArray.
+ * \param NDArray object.
+ * \return size of the memory.
+ */
+inline size_t GetMemSize(NDArray& narr) {
+  size_t size = 1;
+  for (tvm_index_t i = 0; i < narr->ndim; ++i) {
+    size *= static_cast<size_t>(narr->shape[i]);
+  }
+  size *= (narr->dtype.bits * narr->dtype.lanes + 7) / 8;
+  return size;
+}
+
+/*!
+ * \brief Get the input alloc mem size.
+ * \param input_id The input id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetInputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetInputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Get the output alloc mem size.
+ * \param input_id The output id to query the mem size.
+ * \return The memory size.
+ */
+size_t TVMRunner::GetOutputMemSize(std::string input_id) {
+  LOG(INFO) << "TVMRunner::GetOutputMemSize:" << input_id;
+
+  NDArray in_arr = r_graph_handle.GetFunction("get_output")(input_id);
+  auto ssize = GetMemSize(in_arr);
+
+  return ssize;
+}
+
+/*!
+ * \brief Set the model inputs from npz file.
+ * \param inputfile the npz file from where we read input tensor data.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string inputfile) {
+  LOG(INFO) << "TVMRunner::SetInput (Numpy):" << inputfile;
+  cnpy::npz_t npz_input = cnpy::npz_load(inputfile);
+
+  for (auto& elem : mInfo.input_info) {
+    LOG(INFO) << "Set Numpy Input for :" << elem.first;
+    NDArray in_arr = r_graph_handle.GetFunction("get_input")(elem.first);
+    auto ssize = GetMemSize(in_arr);
+
+    if (npz_input.find(elem.first) != npz_input.end()) {
+      in_arr.CopyFromBytes(npz_input[elem.first].data<char>(), ssize);
+    } else {
+      LOG(WARNING) << "Couldn't find input " << elem.first << " in npy input file";
+    }
+  }
+
+  return 0;
+}
+
+/*!
+ * \brief Set the model input from the given binary buffer.
+ * \param input_id input node name.
+ * \param raw_input binary input buffer to copy over input NDArray.
+ * \param 0 on success else error code.
+ */
+int TVMRunner::SetInput(std::string input_id, char* raw_input) {
+  LOG(INFO) << "TVMRunner::SetInput (Raw)";
+  NDArray in_arr = r_graph_handle.GetFunction("get_input")(input_id);
+  auto ssize = GetMemSize(in_arr);

Review Comment:
   ```suggestion
   auto ssize = GetInputMemSize(input_id);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [tvm] junrushao commented on a diff in pull request #13791: [TOOL][NATIVE] Android native application for deploy and run

[GitBox <gi...@apache.org>]

junrushao commented on code in PR #13791:
URL: https://github.com/apache/tvm/pull/13791#discussion_r1071518805


##########
.gitmodules:
##########
@@ -19,3 +19,6 @@
 [submodule "3rdparty/OpenCL-Headers"]
 	path = 3rdparty/OpenCL-Headers
 	url = https://github.com/KhronosGroup/OpenCL-Headers.git
+[submodule "3rdparty/cnpy"]
+	path = 3rdparty/cnpy
+	url = https://github.com/rogersce/cnpy.git

Review Comment:
   I was generally less aggressive when introducing new dependency. It doesn't mean I don't like it in this case, but do you think we could use TVM's native NDArray saving methods instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org