You are viewing a plain text version of this content. The canonical link for it is here.

Posted to discuss-archive@tvm.apache.org by sho via Apache TVM Discuss <no...@discuss.tvm.ai> on 2021/11/26 11:09:47 UTC

[Apache TVM Discuss] [microTVM] Several questions about micro TVM

I'm working for a MCU vendor, and thinking about using micro tvm to deploy some ML models on MCUs.
However, I'm a bit confused with the concepts of standalone execution.

We, as one of MCU vendors, would like to see standalone execution solely on MCUs using micro tvm.
The image below is the concept of standalone execution found in the official doc.

![211125f|309x388](upload://245oDb5b2hSgEVqfIP7BgXJ3GAW.png)

Here I have some questions regarding this.

1. Where is 'micro tvm' in this image? I'm not sure what 'micro tvm' exactly means, and how it's involved in the process of generating binary for MCUs. If 'micro tvm' means the mechanism or something to generate binary and doesn't mean the code or binary that will be stored in MCUs, then what is the one people call 'micro tvm runtime'?

2. Also I'm a bit confused with the difference between tvm and micro tvm. tvm can also generate the same components (Simplified Parameters, Compiled Operators and Graph JSON) right? I'd like to know the difference of the artifacts between by tvm and micro tvm.

3. According to the presentation below:
https://docs.google.com/presentation/d/1Iryuls3D2jv-6__i8CTR-xf718jT31FPI_3nWCXfpeg/edit#slide=id.gd7c0682cbc_0_76
the difference between compilation with AOT and without AOT is the generated C code. To my understanding, AOT enables us to remove Graph Runtime and Graph JSON from the lists stored in MCUs, and instead another kind of C code will be generated which kind of combine compiled operators, Graph Runtime and Graph JSON.
All we need to run inference is this C code(Compiled Operators) , Simplified Parameters, Inputs, device initialization code and application code(those in main). We just download these into MCUs. Is my understanding correct?

4. Regarding the test file: tvm/tests/micro/zephyr/test_zephyr.py
https://github.com/apache/tvm/tree/main/tests/micro/zephyr
As the README suggests, I ran
```
$ cd tvm/apps/microtvm/
$ poetry lock && poetry install && poetry shell
$ cd tvm/tests/micro/zephyr
$ pytest test_zephyr.py --zephyr-board=qemu_x86
```
But I got errors like below.

```
ImportError while loading conftest '/home/ubuntu/workspace/tvm/conftest.py'.
../../../conftest.py:20: in <module>
import tvm
../../../python/tvm/__init__.py:26: in <module>
from ._ffi.base import TVMError, __version__
../../../python/tvm/_ffi/__init__.py:28: in <module>
from .base import register_error
../../../python/tvm/_ffi/base.py:71: in <module>
_LIB, _LIB_NAME = _load_lib()
../../../python/tvm/_ffi/base.py:51: in _load_lib
lib_path = libinfo.find_lib_path()
../../../python/tvm/_ffi/libinfo.py:146: in find_lib_path
raise RuntimeError(message)
E RuntimeError: Cannot find the files.
E List of candidates:
E /home/ubuntu/.cache/pypoetry/virtualenvs/microtvm-vg_j_zxI-py3.8/bin/libtvm.so
E /home/ubuntu/.poetry/bin/libtvm.so
E /usr/local/sbin/libtvm.so
E /usr/local/bin/libtvm.so
E /usr/sbin/libtvm.so
E /usr/bin/libtvm.so
E /usr/sbin/libtvm.so
E /usr/bin/libtvm.so
E /usr/games/libtvm.so
E /usr/local/games/libtvm.so
E /snap/bin/libtvm.so
E /home/ubuntu/workspace/tvm/python/tvm/libtvm.so
E /home/ubuntu/workspace/libtvm.so
E /home/ubuntu/.cache/pypoetry/virtualenvs/microtvm-vg_j_zxI-py3.8/bin/libtvm_runtime.so
E /home/ubuntu/.poetry/bin/libtvm_runtime.so
E /usr/local/sbin/libtvm_runtime.so
E /usr/local/bin/libtvm_runtime.so
E /usr/sbin/libtvm_runtime.so
E /usr/bin/libtvm_runtime.so
E /usr/sbin/libtvm_runtime.so
E /usr/bin/libtvm_runtime.so
E /usr/games/libtvm_runtime.so
E /usr/local/games/libtvm_runtime.so
E /snap/bin/libtvm_runtime.so
E /home/ubuntu/workspace/tvm/python/tvm/libtvm_runtime.so
E /home/ubuntu/workspace/libtvm_runtime.so
```
After successfully running the script, am I supposed to get all the C code to build and download into MCUs?

5. Another script for AOT
tvm/python/tests/relay/aot/test_crt_aot.py
(https://github.com/apache/tvm/blob/main/tests/python/relay/aot/test_crt_aot.py)

There is no information on the environments so I didn't even run the script. But this script generates the C code that can be compiled by let's say arm v7?
At this moment, can we generate C code by AOT, make and customize some application using the generated C code?
In case of TFL micro, we can create a project in IDEs, pull some C code from TFL micro repository and include them in the project(C runtime), convert some tflite models into a large C array, TFL micro loads this array before running inference. This is how we can run inference using TFL micro and there is no need for MCUs connected to the host PC.
I just want to do the same thing by AOT. Is it possible for the moment, and is there any sample of it?

---
[Visit Topic](https://discuss.tvm.apache.org/t/several-questions-about-micro-tvm/11569/1) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/e4714acba7444a187f515b692883dc5f3d858bc9cbdec9c19fc0f5f3ef87e556).

[Apache TVM Discuss] [microTVM] Several questions about micro TVM

Posted by sho via Apache TVM Discuss <no...@discuss.tvm.ai>.


Thank you leandron, for letting the developers know this question!





---
[Visit Topic](https://discuss.tvm.apache.org/t/several-questions-about-micro-tvm/11569/3) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/b23208261b69bd9a698862cc21d35850793e775b39fb50d34e385b9a2beaa59e).

[Apache TVM Discuss] [microTVM] Several questions about micro TVM

Posted by "Leandro Nunes (Arm) via Apache TVM Discuss" <no...@discuss.tvm.ai>.


cc @areusch, @gromero, @grant-arm for visibility





---
[Visit Topic](https://discuss.tvm.apache.org/t/several-questions-about-micro-tvm/11569/2) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/f402e9c78ccc41488dd47bb5d8c840f3cc86b573453f9ddcd93106b347d8e965).

[Apache TVM Discuss] [microTVM] Several questions about micro TVM

Posted by Andrew Reusch via Apache TVM Discuss <no...@discuss.tvm.ai>.


Sorry for the long delay!

[quote="sho, post:7, topic:11569"]
[quote="areusch, post:6, topic:11569"]
There are actually two GraphExecutor implementations in TVM: [one ](https://github.com/apache/tvm/blob/main/src/runtime/graph_executor/graph_executor.cc) for the C++ Runtime and [one ](https://github.com/apache/tvm/blob/main/src/runtime/crt/graph_executor/graph_executor.c) for the C Runtime. Sorry for this confusion.
[/quote]

Oh, ok. I should have noticed that GraphExecutor written in C is in crt. Looking though the contents of tvm/src/runtime/crt, I can create a C project on IDE with the source code here and the outputs from tvm.relay.build + SoC(or MCU) initialization code right?
[/quote]

Yeah the embedding of CRT can be a little confusing. We should probably think about whether we should reorganize src/runtime at some point. Yes, you should be able to create a C project in your IDE as you described.

[quote="sho, post:7, topic:11569"]
But this might not always be the case since as for Host-driven execution, which requires rpc server/client communication, the scripts for this are in ... and they are written in C++ right?
[/quote]

Yeah you're right here. We need to do a rewrite there into C, but we need to invest more heavily in C-based RPC unit testing first.





---
[Visit Topic](https://discuss.tvm.apache.org/t/several-questions-about-micro-tvm/11569/8) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/5f2ac53760a83f8bcd5fb00b3cd7d0b0bf0e596f17d273100ab8a1905903e229).

[Apache TVM Discuss] [microTVM] Several questions about micro TVM

Posted by sho via Apache TVM Discuss <no...@discuss.tvm.ai>.

Thank you very much for your answers.

[quote="areusch, post:6, topic:11569"]
There are actually two GraphExecutor implementations in TVM: [one ](https://github.com/apache/tvm/blob/main/src/runtime/graph_executor/graph_executor.cc) for the C++ Runtime and [one ](https://github.com/apache/tvm/blob/main/src/runtime/crt/graph_executor/graph_executor.c) for the C Runtime. Sorry for this confusion.
[/quote]

Oh, ok. I should have noticed that GraphExecutor written in C is in crt. Looking though the contents of tvm/src/runtime/crt, I can create a C project on IDE with the source code here and the outputs from tvm.relay.build + SoC(or MCU) initialization code right?

[quote="areusch, post:6, topic:11569"]
could you clarify this sentence? I think you meant to say something different in the second “micro TVM.”
[/quote]

Apologies for my mistakes and confusing you. I wanted to say

"I understood that the key difference is micro TVM uses C runtime and TVM uses C++ runtime, and the latter may not work on MCUs for the moment."

But this might not always be the case since as for Host-driven execution, which requires rpc server/client communication, the scripts for this are in

https://github.com/apache/tvm/tree/main/src/runtime/rpc
https://github.com/apache/tvm/tree/main/src/runtime/crt/microtvm_rpc_common
https://github.com/apache/tvm/tree/main/src/runtime/crt/microtvm_rpc_server

and they are written in C++ right?

I think before I dive into micro TVM, I should begin with TVM so that I will have clear understanding of the difference between TVM and micro TVM.

[quote="areusch, post:6, topic:11569"]
We’ll be releasing some roadmaps for microTVM and TVM as a whole ~next month.
[/quote]

Thank you for your release information. I'm looking forward to its release.

[quote="areusch, post:6, topic:11569"]
Yes, the microtvm-blogpost-eval does explain the approach, but unfortunately it’s a bit out of date. We should have a new tutorial at some point soon that uses Project API. I’d also suggest you look for the Arduino tutorial coming up at TVMCon. Apologies for lacking documentation here.
[/quote]

Ok, that's fine as long as it works anyway. Also I'll try to run the Arduino tutorial when it's published. Thank you for your suggestion.

---
[Visit Topic](https://discuss.tvm.apache.org/t/several-questions-about-micro-tvm/11569/7) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/cd32ac257e4936b19404ad97c631afcdc2463496989062250bd77a8cfe1998c2).

[Apache TVM Discuss] [microTVM] Several questions about micro TVM

Posted by Andrew Reusch via Apache TVM Discuss <no...@discuss.tvm.ai>.


[quote="sho, post:5, topic:11569"]
So the Graph Runtime works on top of C Runtime? Could you please tell me where the C Runtime actually is? I found the link below but it seems that graph_executor is written in C++.
[/quote]

There are actually two GraphExecutor implementations in TVM: [one](https://github.com/apache/tvm/blob/main/src/runtime/graph_executor/graph_executor.cc) for the C++ Runtime and [one](https://github.com/apache/tvm/blob/main/src/runtime/crt/graph_executor/graph_executor.c) for the C Runtime. Sorry for this confusion.

[quote="sho, post:5, topic:11569"]
I understood that the key difference is micro TVM uses C runtime and micro TVM uses C++ runtime, and the latter may not work on MCUs for the moment.
[/quote]

could you clarify this sentence? I think you meant to say something different in the second "micro TVM."

[quote="sho, post:5, topic:11569"]
AOT Executor is the C code that can be downloaded onto MCUs?
[/quote]

Yes.

[quote="sho, post:5, topic:11569"]
I’d like to know what capabilities you are adding to AOT.
[/quote]

We'll be releasing some roadmaps for microTVM and TVM as a whole ~next month.

[quote="sho, post:5, topic:11569"]
Also, I’m curious about how you can execute GraphExecutor + GraphJSON on MCUs for the moment. I know it’s difficult for MCUs due to memory constraints, but I’d like to see how you deploy standalone GraphExecutor + GraphJSON to run inference on MCUs without communicating with the host PC. Should I be able to see it in the tutorial below?
[/quote]

Yes, the microtvm-blogpost-eval does explain the approach, but unfortunately it's a bit out of date. We should have a new tutorial at some point soon that uses Project API. I'd also suggest you look for the Arduino tutorial coming up at TVMCon. Apologies for lacking documentation here.





---
[Visit Topic](https://discuss.tvm.apache.org/t/several-questions-about-micro-tvm/11569/6) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/e943810b9b9b6b14809671bb7a72ab62c565c5c878adf0669cd92a62a6e7d823).