You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tvm.apache.org by Andrew Reusch via Apache TVM Discuss <no...@discuss.tvm.ai> on 2021/02/12 03:17:47 UTC

[Apache TVM Discuss] [Development/RFC] [RFC] [µTVM] Model Library Format

# Model Library Format

## Background

TVM's build process for imported models centers around `tvm.relay.build`, a function which produces a 3-tuple `(graph_json, lib, params)`. The inference workflow then diverges depending on how the user wants to use the compiled artifacts:

- If the build targets the c++ runtime and uses the `llvm` backend...
- and the user wants to run in the same Python instance used to compile: the user can directly instantiate a GraphRuntime instance.
- and the user wants to transfer the model to another Python runtime instance without cross-compiling: the user can call `lib.export_library()`, and store `graph_json` and `params` in some ad-hoc way. Then, `tvm.runtime.load_module()` can recreate `lib` in the new runtime instance.
- and the user wants to transfer the model to another Python runtime instance with cross-compiling: the same procedure as above, but pass `fcompile` to `export_library` to specify the cross-compiler.
- If the build targets the c++ runtime and uses the `c` backend...
- and the user wants to run the model with Python on similar architecture: the user must compile the produced `c` files to produce an artifact similar to the one produced by `lib.export_library()`. Then, they can load and run the library following the procedure above. When saving and loading from the same instance (so `graph_json` and `params` are not a consideration), this process is handled invisibly by `loadfile_tar`.
- and the user wants to run the model with Python on different architecture: same procedure as above, but with a cross-compiler.
- and the user wants to run the model with a different frontend language: same procedure as above, but the user must translate `graph_json` and `params` to a format suitable for the other language
- If the build targets the c runtime...
- and the user wants to run the model with TVM in Python: not supported — Python supports C++ runtime only.
- and the user wants to run standalone: compile with `-system-lib`, store the library in a `.tar` with `export_library()`, store `params` and `graph_json` to disk in an ad-hoc way, unpack the tar and integrate all pieces into a standalone project. A small `main` is needed to launch the C runtime, load the model and parameters, and run inference. See `apps/bundle_deploy`.

In all cases *except* the first (compile and run in the same TVM instance), the user needs to serialize the `tvm.relay.build` 3-tuple before doing anything else. However, TVM provides no common function to handle this—it only directly handles serializing the compiled library. The user is left to store the parameters and runtime configuration (e.g. `graph_json`) in a way that suits the task at hand. This discrepancy means that all the automation that consumes TVM artifacts from disk is always hand-written and specific to the situation.

On microTVM, we are preparing to introduce a Project-level API, implementations of which a) live in separate codebases from `tvm` and b) build firmware images from the `tvm.relay.build` artifacts. Due to this burden, the API needs to specify how all artifacts from `tvm.relay.build` are placed on-disk.

To prepare for this API, we propose Model Library Format, a standard on-disk format for microTVM artifacts. microTVM primarily expects users to use the `c` or `llvm` backends with a cross-compiler, and build results may contain BYOC artifacts as well. As a secondary goal to this RFC, we make some considerations such that Model Library Format could be re-used as the standard on-disk format produced by `tvmc`.

## Goals

- Describe a standard way to serialize microTVM artifacts for use in downstream automation to compile them into firmware
- Describe how to implement a load API such as `tvm.runtime.load_module() -> GraphRuntimeFactory`.
- Make considerations to accommodate other runtimes such as AOT and VM.

## Non-Goals

- Immediately change the `tvmc` output format to Model Library Format for non-µTVM uses. The initial implementation is focused to microTVM only.
- Decide how to serialize compilation flows unrelated to microTVM

## Model Library Format

Model Library Format is a tar-archived directory tree. A sketch is as follows:

```bash
/
README.md - A short standardized README for new users plus human-readable metadata.json
metadata-<n>.json - Overall metadata describing this artifact; version <n>
crt/ - The content of standalone_crt from TVM build/
Makefile
include/
...
src/
...
codegen/ - Stores generated libraries in source or binary form
host/ - Generated code for target_host
lib/ - Generated binary object files
aot.o - Future home of AOT runtime generated code
devc.o - C++ MetadataModule artifact, unused in µTVM. Should get deleted.
lib0.o - LLVM module
lib1.o - LLVM CRT Metadata Module
src/ - Generated C source
devc.c - C++ MetadataModule artifact, unused in µTVM. Should get deleted.
lib0.c - C module
lib1.c - C CRT Metadata module
target_key/ - Additional directories for code which should get compiled for use on a target.
parameters/ - Stores simplified parameters
<model_name>.bson - BSON-serialized runtime parameters (optional)
<model_name>.params - tvm.relay._save_params format (always present)
<model_name>.json - JSON-serialized parameters (optional)
relay.txt - text representation of the relay model compiled, if built from Relay
runtime-config/ - Stores runtime configuration.
aot/ - AOT runtime config
(tbd)
graph/ - Graph runtime config
graph.json - Graph runtime JSON
```

### metadata.json

The metadata file contains machine-parseable data describing the build. It also contains model-level information that is easier (right now) to parse as a single JSON document rather than split into many smaller purpose-specific files.

Following is a proposed schema:

```bash
{
"version": 1, // version of this document.
"model_name": "<model_name>", // model name, (passed as mod_name= to tvm.relay.build).
"export_datetime_utc": "%Y-%m-%d %H:%M:%S" // Time of export, in UTC.
"memory": {}, // configured memory map (see Memory Map)
"target": "", // TVM target string used to compile this artifact
"runtimes": ["graph"], // The runtimes that can launch this model.
}
```

### Memory Map

In v1, the Memory Map will describe the buffers allocated by the GraphRuntime. As the memory planner is improved, this data structure will be expanded. Following is the schema for the "memory" key in v1:

```bash
[
{
"storage_id": <n>, // storage_id of the buffer, allocated by GraphRuntime
"size_bytes": <n>, // size of this buffer, in bytes
"input_binding": "" // when bound to a model input, the name of that input
},
// Additional entries
]
```

## Building a Model Library Format

Here is the process by which TVM creates a Model Library Format from `[tvm.relay.build](http://tvm.relay.build)` artifact. Here, `graph_json`, `lib`, and `params` are the 3-tuple returned and `target` is the TVM target. mkdir is assumed.

1. If `target` contains `--runtime=crt`, copy `$tvm_root/build/standalone_crt` to `./crt`.
2. Populate `./codegen` by calling `lib.export_library()`, which should:
1. Collect all Modules that execute on the host and pass to `fcompile`. At present, these are those with `type_key()` of `c` or `llvm`. When the `c` target is used, `fcompile` should copy the generated files into `./codegen/host/src` instead of generating a `.tar`.
2. (TODO, but not as a result of this RFC) Group the non-host modules by target_type (except that ext_dev target_types should be expanded to a unique key per BYOC). Save each generated module into a file underneath `./codegen/<target_type>`.
3. Populate `./parameters` .
- Produce `<model_name>.params` with `tvm.relay._save_params`.
- Produce `<model_name>.json` with TBD (there doesn't seem to be a standard in TVM, so I guess we'll have to propose one)
4. Produce `relay.txt` with `IRModule.get_source`
5. Produce `./runtime-config` as follows:
- for GraphRuntime: save `graph.json` to `./runtime-config/graph/graph.json`
- for VM: TBD
- for AOT: TBD
6. Produce `metadata-<n>.json` by building the required data structure and serializing to JSON.

Finally, the entire directory tree should be packaged into a TAR file with `.model-lib` extension for easy transmission.

## Implementation in TVM

The implementation of this RFC will initially consist of the following:

1. Adding a new function, `tvm.runtime.Module#export_model_library_format`. This function implements the above procedure for runtimes which use the `c` backend.
2. Placing the state necessary to implement `export_model_library_format` into GraphRuntimeCodegenModule, and making it accessible from Python.
3. Adding `loadfile_model_lib` which allows loading `tvm.runtime.GraphRuntimeFactoryModule` from the file produced by `export_model_library_format`.
4. Adding unit tests and changing apps/bundle_deploy to use this format as an example.

Following implementation of this RFC, another RFC (Project-level API for µTVM projects) will be submitted explaining how we intend to refactor the current interaction between TVM and µTVM runtime projects to allow for better portability. Also, `tvmc` will begin creating Model Library Format for `--runtime=c` targets.

## µTVM Use Cases

Here I briefly walk through some µTVM use cases of Model Library Format to consider whether it's a net improvement.

### Building Host-Driven Firmware (µTVM)

At present, µTVM builds host-driven firmware (GraphRuntime instantiated on the host) as follows:

1. The user instantiates an implementation of `tvm.micro.Compiler`.
2. TVM invokes `tvm.micro.Compiler#library` to compile each CRT sub-library and the code in `./codegen/host`.
3. TVM invokes `tvm.micro.Compiler#binary` to build a binary firmware image including each library.

Following implementation of this change, the compilation flow will remain the same, but the CRT sources used will be taken from the Model Library Format tree.

### Host-Driven Inference

At present, this is done from within the same Python script as called `[tvm.relay.build](http://tvm.relay.build)` since it's easier to keep all of the state in memory. This can be done with a separate `python` invocation, but there is no standard function to load all of the state necessary, so it's ad-hoc. Following this change, the GraphRuntimeFactoryModule can be loaded using `tvm.runtime.load_module`, so it will be much easier to reconstruct the state needed for host-driven inference.

### Building Standalone Firmware (e.g. `apps/bundle_deploy`)

Currently, `apps/bundle_deploy` invokes a custom Python script which produces artifacts in `apps/bundle_deploy/build`. After this RFC, `apps/bundle_deploy/build_model.py` will produce Model Library Format artifacts for the C-runtime compatible artifacts.

For `apps/bundle_deploy`, the Makefile will be updated to reference the artifacts in standard locations. In the future, it will be possible to write a standard script to ingest generated code as a library into project build systems.

## Future Work

We expect to make changes as future considerations are made in Model Library Format. Each time a change is made, the version number will be incremented. Here are some sketches of future topics that could be tackled.

### Contexts

In heterogeneous execution, this object will describe the various DLContexts that TVM expects to be configured on the device. This RFC doesn't seek to fully describe this key—heterogeneous execution is a future goal of µTVM, and until something more concrete is proposed there, this key will just contain an entry for `DLContext(kDLCPU, 0)`.

Here is a strawman:

```bash
"contexts": [
{
"device_type": "cpu",
"device_id": 0,
},
{
"device_type": "ext_dev",
"device_id": 0,
"compiler": "accel_compiler_key",
"config": {
// device-specific config, populated by BYOC
},
},
], // configured DLContext (see DLContext configuration)
```

### Models Targeted to the C++ Runtime

Models targeted to the C++ runtime have very similar structure to those targeted at the C runtime. The main difference is in how non-`c` and non-`llvm` ("non-DSO-Exportable") modules are packaged.

The C++-runtime places all modules in a single shared library like a "fat binary." At load time, it expects to find a constant `__tvm_dev_mblob` which contains concatenated `Module#save` from all of these modules. It then invokes a `runtime.module.loadbinary_<type_key>` for each Module in `__tvm_dev_mblob`.

In the C runtime, non-DSO-Exportable modules are typically created from BYOC flows and are meant to be executed by accelerators. Because RAM is typically quite precious on µC, the C runtime intends to make such generated BYOC code available to the downstream firmware build at compile time. Modules are grouped by `target_type` one file is generated per Module containing `Module#save` .

It's possible that both approaches could be taken for C++ runtime to allow pre-compilation of Modules. However, the simplest and most likely way to move forward would be to create `./codegen/<model_name>.so` and avoid creating subdirectories. When the `c` backend is used with the C++ runtime, `./codegen/host/src` could still be created, or the `.tar` could be placed in `./codegen/<model_name.tar>`.

@tqchen @gromero @leandron @manupa-arm @mdw-octoml @jroesch @mjs @liangfu

---
[Visit Topic](https://discuss.tvm.apache.org/t/rfc-tvm-model-library-format/9121/1) to respond.

You are receiving this because you enabled mailing list mode.

To unsubscribe from these emails, [click here](https://discuss.tvm.apache.org/email/unsubscribe/9b28c1eacbbbfb80ca1d98bb194e3ba122566233b3e5a6e0595da53357363f67).