You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2021/03/25 15:04:20 UTC
[GitHub] [tvm] stoa opened a new pull request #7742: Contributing the STM32 port
stoa opened a new pull request #7742:
URL: https://github.com/apache/tvm/pull/7742
We propose to contribute the STM32 port of the TVM.
The STM32 are mainly bare-metal boards and often use-cases do not suppose using an OS: Linux, embed,RTOS,etc.
The STM32 development tools (ex, CubeMX) build standalone applications (no OS) that can be flashed to the board and executed.
In this TVM port, the TVM is used to compile AI networks into a C code compliant with the STM32 AI API.
In particular, the TVM is used to build a C module (kernels library + JSON graph + params); then the stm32.emitter.py generates the C implementation from the TVM module's JSON graph and writes out a C implementation of the kernels.
The generated C code can then be included as part of a larger STM32 application, and managed within a standard STM32 application development flow.
This contribution includes:
- a python/tvm/contrib/stm32 directory with scripts necessary for compiling and communication with the STM32 board
- a src/runtime/stm32 directory that includes runtime support used by the STM32 applications
- an apps/stm32 README and examples running in the STM32 DISCOVERY H747 board.
Tested on development board with several models including mnist, inceprion, squeezenet, mobilenet, yolo, and a number of proprietary models.
This contribution does not modify any of the original TVM files.
@areusch
@tqchen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r629731022
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
hi @stoa, sorry for the delay again. i'll take a look at the parts of this PR outside of `tests/crt/contrib/stm32`.
> These would be good candidates for including with the crt_backend_api.c. Therefore, moving the TVMAPISetLastError() and the TVMGetLastError() there from the crt_runtime_api.c, and moving the TVMBackendParallelLaunch(), TVMBackendRegisterSystemLibSymbol() out to crt_runtime_api.c.
I think that makes sense to me. `crt_backend_api.c` should be able to be largely reused across platforms.
> Currently, the TVMBackendAllocWorkspace does some error handling. I would propose to leave the error handling to the caller, while the TVMBackendAllocWorkspace would just return NULL in case of error.
The reason is that the error handling and logging are platform-specific and even application-specific - eg. malloc does not do error handling, for example.
There's a couple concerns I have here:
1. We'd have to generate error-checking code in operator implementations, since that is where `TVMBackendAllocWorkspace` is called from. It's hard to generate macros such as `CHECK_EQ` there.
2. Returning NULL would obscure the error code. I added the error code for traceability--in my experience, it's often valuable to have a trace of the execution path that trips an error code. On embedded, it can be quite hard to get these traces particularly in production. Therefore, having the error code helps to debug without necessarily needing to have a debugger attached. We could explore changing the signature of TVMBackendAllocWorkspace, but it could break people
> If we were to reduce the crt_backend_api.c to its minimal form as I have described above, perhaps it makes sense to have the application implement these few methods directly, i.e. no need to include the tvm/crt/crt_backend_api.c implementation, only the crt_backend_api.h header as interface ?
Potentially for the memory allocation function, but I'm not so sure about the rest. The rest are for interop with the runtime. As we add support for e.g. accelerators or better memory planning here, I could see these implementations becoming a bit more complex. I'd prefer to stick with the application hook `TVMPlatformMemoryAllocate` for now, if that's the only divergence you can see per-application.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-873088794
@stoa let me discuss it with @tqchen a bit and get back to you. in general i'm okay with committing things in multiple stages, but this is quite a core interface so i'd not like to duplicate functionality. i think to some degree this is an organizational choice made by the codebase.
at a high level, it makes sense to me that the conceptual c_runtime_api is a superset of c_backend_api. however, a confounding factor is that compiled functions can call PackedFunc e.g. defined inside the tvm codebase, and this pattern is prolific in the non-micro use case (C++ runtime).
one question for you: do we need to block on this to merge the STM32 port? I think the main sticking point is TVMAPISetLastError, and I haven't been using that in the micro implementations (instead, I just return an error code to match [include/tvm/runtime/crt/error_codes.h](https://github.com/apache/tvm/blob/main/include/tvm/runtime/crt/error_codes.h) in place of the `-1` or `0` return value from the functions defined in `c_backend_api`. This serves as a better debugging aid without requiring a lot of strings in flash.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-872796133
@areusch
Hello Andrew
I cannot just move the TVMLastError to c_backend because then the c_runtime will depend on c_backend and the c_backend already depends on c_runtime. From our previous exchange we agreed that we want c_runtime depend on c_backend, so I moved the minimum of code from c_runtime to c_backend that had to be moved in order to respect this dependency.
I can move these changes to a separate PR, even give it more work, because I really moved the minimum possible. However, I would like to close this PR first. Therefore, I am proposing back what I have suggested earlier: I duplicate the TVMLastError functionality inside out platform-specific implementation (we do not link c_runtime_api.c) and I'd take care of aligning the c-backend vs c_runtime functionality after that. How about this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r626071377
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
Thanks Andrew, @areusch
I am aligning on your comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r716190042
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ #
+ # Constants:
+ #
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ self.activations_static = include_activations
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def __tensor_is_output(self, nid, idx):
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def __get_tensor_from_node(self, nid, idx):
+ # 'eid' is index into the dltype', 'shape', etc.
+ eid = self.node_row_ptr_[nid] + idx
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size_bytes(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+ buffer_list_ = {}
+
+ nid = 0
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self.__tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+ for output in self.outputs_:
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
Review comment:
I prefer to keep a separate data structure for the emitter. Generally, this data structure is not necessarily equal to the TVM graph_['nodes']. For example, I may decide to not include OPs that are a 'noop' in the code, etc. This is done on purpose to simplify development. No problem maintaining - the structure is built in a single place from the graph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-927609563
@areusch
How are you, Andrew.
Done. Please check.
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-869893211
hey @stoa that look like a flake to me. can you push an empty commit or git commit --amend to retrigger?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-821673840
@areusch
Do you have a pointer on how to deal with this:
clang-format lint error found. Consider running clang-format-10 on these files to fix them.
?
Some commands from https://tvm.apache.org/docs/contribute/pull_request.html fail with this:
Running './tests/lint/git-clang-format.sh origin/main' inside tlcpack/ci-lint...
Unable to find image 'tlcpack/ci-lint:latest' locally
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r663522121
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
Review comment:
nit: can't see where this is used
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? ((AI_INTQ_INFO_LIST_FLAGS(AI_BUFFER_META_INFO_INTQ(meta_)) & \
+ AI_BUFFER_META_FLAG_ZEROPOINT_U8) \
+ ? AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_u8, pos_) \
+ : AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_i8, pos_)) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INIT(flags_, intq_info_) \
+ { .flags = (flags_), .intq_info = AI_PACK(intq_info_) }
+
+#define AI_BUFFER_SIZE(buf_) (((buf_)->width) * ((buf_)->height) * ((buf_)->channels))
+
+#define AI_BUFFER_BYTE_SIZE(count_, fmt_) ((((count_)*AI_BUFFER_FMT_GET_BITS(fmt_)) + 4) >> 3)
+
+/*!
+ * @enum buffer formats enum list
+ * @ingroup ai_platform
+ *
+ * List of supported ai_buffer format types.
+ */
+enum {
+ AI_BUFFER_FORMAT_NONE = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_NONE, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_FLOAT = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_FLOAT, 1, 1, 32, 0),
+
+ AI_BUFFER_FORMAT_U8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 0),
+ AI_BUFFER_FORMAT_U16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 0),
+ AI_BUFFER_FORMAT_S8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 0),
+ AI_BUFFER_FORMAT_S16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 0),
+
+ AI_BUFFER_FORMAT_Q = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 0, 0),
+ AI_BUFFER_FORMAT_Q7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 7),
+ AI_BUFFER_FORMAT_Q15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_UQ = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_UQ7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 7),
+ AI_BUFFER_FORMAT_UQ15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_BOOL = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_BOOL, 0, 0, 8, 0),
+};
+
+/******************************************************************************/
+
+/* printf formats */
+#define SSIZET_FMT "%" PRIu32
+#define AII32_FMT "%" PRId32
+#define AIU32_FMT "%" PRIu32
+
+typedef uint8_t ai_custom_type_signature;
+
+typedef void* ai_handle;
+
+typedef void (*ai_handle_func)(void*);
+
+typedef float ai_float;
+typedef double ai_double;
+
+typedef bool ai_bool;
+
+typedef uint32_t ai_size;
+
+typedef uintptr_t ai_uptr;
+
+typedef unsigned int ai_uint;
+typedef uint8_t ai_u8;
+typedef uint16_t ai_u16;
+typedef uint32_t ai_u32;
+typedef uint64_t ai_u64;
+
+typedef int ai_int;
+typedef int8_t ai_i8;
+typedef int16_t ai_i16;
+typedef int32_t ai_i32;
+typedef int64_t ai_i64;
+
+typedef uint32_t ai_signature;
+
+/******************************************************************************/
+/*!
+ * @struct ai_intq_info
+ * @ingroup ai_platform
+ * @brief an element of the ai_intq_info_list entry. It reports an array for the
+ * scale and zeropoint values for each buffer. Optional flags are also present
+ */
+typedef struct ai_intq_info_ {
+ INTQ_CONST ai_float* scale;
+ INTQ_CONST ai_handle zeropoint;
+} ai_intq_info;
+
+/*!
+ * @struct ai_intq_info_list
+ * @ingroup ai_platform
+ * @brief list reporting meta info for quantized networks integer support
+ * when size > 1 it means a per channel out quantization
+ */
+typedef struct ai_intq_info_list_ {
+ ai_u16 flags; /*!< optional flags to store intq info attributes */
+ ai_u16 size; /*!< number of elements in the the intq_info list */
+ INTQ_CONST ai_intq_info* info; /*!< pointer to an array of quant info */
+} ai_intq_info_list;
+
+/******************************************************************************/
+/*!
+ * @struct ai_buffer_meta_info
+ * @ingroup ai_platform
+ * @brief Optional meta attributes associated with the I/O buffer.
+ * This datastruct is used also for network querying, where the data field may
+ * may be NULL.
+ */
+typedef struct ai_buffer_meta_info_ {
+ ai_u32 flags; /*!< meta info flags */
+ ai_intq_info_list* intq_info; /*!< meta info related to integer format */
+} ai_buffer_meta_info;
+
+/*!
+ * @struct ai_buffer
+ * @ingroup ai_platform
+ * @brief Memory buffer storing data (optional) with a shape, size and type.
+ * This datastruct is used also for network querying, where the data field may
+ * may be NULL.
+ */
+typedef struct ai_buffer_ {
+ ai_buffer_format format; /*!< buffer format */
+ ai_u16 n_batches; /*!< number of batches in the buffer */
+ ai_u16 height; /*!< buffer height dimension */
+ ai_u16 width; /*!< buffer width dimension */
+ ai_u32 channels; /*!< buffer number of channels */
+ ai_handle data; /*!< pointer to buffer data */
+ ai_buffer_meta_info* meta_info; /*!< pointer to buffer metadata info */
+} ai_buffer;
+
+/*!
+ * @struct ai_platform_version
+ * @ingroup ai_platform
+ * @brief Datastruct storing platform version info
+ */
+typedef struct ai_platform_version_ {
+ ai_u8 major;
+ ai_u8 minor;
+ ai_u8 micro;
+ ai_u8 reserved;
+} ai_platform_version;
+
+/*!
+ * @struct ai_network_report
+ * @ingroup ai_platform
+ *
+ * Datastructure to query a network report with some relevant network detail.
+ */
+typedef struct ai_network_report_ {
+ const char* model_name;
+ const char* model_signature;
+ const char* model_datetime;
+
+ const char* compile_datetime;
+
+ const char* runtime_revision;
+ ai_platform_version runtime_version;
+
+ const char* tool_revision;
+ ai_platform_version tool_version;
+ ai_platform_version tool_api_version;
+
+ ai_platform_version api_version;
+ ai_platform_version interface_api_version;
+
+ ai_u32 n_macc;
+
+ ai_u16 n_inputs;
+ ai_u16 n_outputs;
+ ai_buffer* inputs;
+ ai_buffer* outputs;
+
+ ai_buffer activations;
+ ai_buffer params;
+
+ ai_u32 n_nodes;
+
+ ai_signature signature;
+} ai_network_report;
Review comment:
i'm not sure i see this struct used at all in the codebase. aside from a couple typedefs used in `ai_runtime_api.h`, i'm not sure this file is used. can we either delete it or pare it down considerably? i'm not sure if maybe this file is typically provided by X-Cube, but it would be great to provide a minimal bridge between TVM and X-Cube by just including the typedefs needed for `ai_runtime_api.h`.
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
Review comment:
nit: can't see where this is used
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? ((AI_INTQ_INFO_LIST_FLAGS(AI_BUFFER_META_INFO_INTQ(meta_)) & \
+ AI_BUFFER_META_FLAG_ZEROPOINT_U8) \
+ ? AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_u8, pos_) \
+ : AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_i8, pos_)) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INIT(flags_, intq_info_) \
+ { .flags = (flags_), .intq_info = AI_PACK(intq_info_) }
+
+#define AI_BUFFER_SIZE(buf_) (((buf_)->width) * ((buf_)->height) * ((buf_)->channels))
+
+#define AI_BUFFER_BYTE_SIZE(count_, fmt_) ((((count_)*AI_BUFFER_FMT_GET_BITS(fmt_)) + 4) >> 3)
Review comment:
nit: can't see where this is used
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
Review comment:
nit: can't see where this is used
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? ((AI_INTQ_INFO_LIST_FLAGS(AI_BUFFER_META_INFO_INTQ(meta_)) & \
+ AI_BUFFER_META_FLAG_ZEROPOINT_U8) \
+ ? AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_u8, pos_) \
+ : AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_i8, pos_)) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INIT(flags_, intq_info_) \
+ { .flags = (flags_), .intq_info = AI_PACK(intq_info_) }
+
+#define AI_BUFFER_SIZE(buf_) (((buf_)->width) * ((buf_)->height) * ((buf_)->channels))
+
+#define AI_BUFFER_BYTE_SIZE(count_, fmt_) ((((count_)*AI_BUFFER_FMT_GET_BITS(fmt_)) + 4) >> 3)
+
+/*!
+ * @enum buffer formats enum list
+ * @ingroup ai_platform
+ *
+ * List of supported ai_buffer format types.
+ */
+enum {
+ AI_BUFFER_FORMAT_NONE = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_NONE, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_FLOAT = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_FLOAT, 1, 1, 32, 0),
+
+ AI_BUFFER_FORMAT_U8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 0),
+ AI_BUFFER_FORMAT_U16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 0),
+ AI_BUFFER_FORMAT_S8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 0),
+ AI_BUFFER_FORMAT_S16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 0),
+
+ AI_BUFFER_FORMAT_Q = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 0, 0),
+ AI_BUFFER_FORMAT_Q7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 7),
+ AI_BUFFER_FORMAT_Q15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_UQ = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_UQ7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 7),
+ AI_BUFFER_FORMAT_UQ15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_BOOL = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_BOOL, 0, 0, 8, 0),
+};
+
+/******************************************************************************/
+
+/* printf formats */
+#define SSIZET_FMT "%" PRIu32
+#define AII32_FMT "%" PRId32
+#define AIU32_FMT "%" PRIu32
+
+typedef uint8_t ai_custom_type_signature;
+
+typedef void* ai_handle;
+
+typedef void (*ai_handle_func)(void*);
Review comment:
nit: can't see where this is used
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? ((AI_INTQ_INFO_LIST_FLAGS(AI_BUFFER_META_INFO_INTQ(meta_)) & \
+ AI_BUFFER_META_FLAG_ZEROPOINT_U8) \
+ ? AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_u8, pos_) \
+ : AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_i8, pos_)) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INIT(flags_, intq_info_) \
+ { .flags = (flags_), .intq_info = AI_PACK(intq_info_) }
+
+#define AI_BUFFER_SIZE(buf_) (((buf_)->width) * ((buf_)->height) * ((buf_)->channels))
Review comment:
nit: can't see where this is used
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
Review comment:
nit: can't see where this is used
##########
File path: python/tvm/contrib/stm32/BaseEmitter.py
##########
@@ -0,0 +1,1257 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import tarfile
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """Replace ':' with '_' in names like 'InputImg:0'"""
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def get_tensor_data_name(tensor_name):
+ return tensor_name + ".dltensor.data"
+
+
+def get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+def get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+
+def _emit_tensor_quant(dl_tensor_name, quantization_map, out_c):
+
+ if dl_tensor_name in quantization_map:
+ quantization = quantization_map[dl_tensor_name]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ quant_name = dl_tensor_name + "_quant"
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+
+def emit_tensor_init(dl_tensor_name, tensor, quantization_map, out_c):
+ """Emit tensor definition."""
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ if "device" in tensor:
+ device = tensor["device"]
+ else:
+ device = "kDLCPU"
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ _emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ if quantization_map is not None:
+ quant_name = _emit_tensor_quant(dl_tensor_name, quantization_map, out_c)
+ else:
+ quant_name = None
+
+ #
+ # Contents
+ #
+ out_c.write(f"AI_ALIGNED({8}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n")
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ out_c.write(f" .device = {{ {device}, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+
+class BaseEmitter:
+ """Base code emitter class."""
+
+ def __init__(
+ self, model_name, include_activations=True, include_inputs=True, include_outputs=True
+ ):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Fix the model name
+ #
+ model_name = re.sub("[^0-9a-zA-Z_]+", "_", model_name)
+ model_name = model_name.lower()
+ self.model_name_ = model_name
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.graph_ = None
+ self.params_ = {}
+ self.lib_ = None
+
+ def _parse_model(self):
+ """Parse the module graph. Build internal data structures."""
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ elif key == "metadata":
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ def parse_library_format(self, model_library_format_path):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_name :
+ Name for this model.
+
+ model_library_format_path :
+ The module to parse
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ self.params_[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+
+ self.graph_ = graph_dict
+ self.lib_ = src_files
+
+ self._parse_model()
+
+ def parse_model(self, module):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_name :
+ Name for this model.
+
+ module : TVM module
+ The module to parse
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ self.params_[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.lib_ = module.get_lib()
+
+ self._parse_model()
+
+ def _tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def _get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ size = get_tensor_size(dims, dltype)
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ def _get_node_input_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self._get_node_input_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def compute_data_placement(self):
+ """Compute inputs, outputs, weight, activation sizes."""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+ offset = 0
+
+ #
+ # Order is important := the order of how the weights will be
+ # written to memory.
+ #
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ #
+ # May not find
+ #
+ if nid == len(self.nodes_):
+ continue
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self._get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self._tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self._tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self._get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self._get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self._get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ def _emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ def _emit_activation_init(self, name, out_c):
+ """Emits buffer initialization code for activation tensors."""
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self._emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self._emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self._emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ def _emit_params_init(self, name, out_c):
+ """Emits buffer initialization code for params tensors."""
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ def emit_init(self, out_c):
+ """Emits buffer initialization code."""
+ #
+ # {name}_configure_activations
+ #
+ self._emit_activation_init(self.model_name_, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self._emit_params_init(self.model_name_, out_c)
+
+ def emit_operator_proto(self, node, out_c):
+ """Emit operator C prototype."""
+
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ if func_name == "__nop":
+ return
+
+ if func_name == "__copy":
+ return
+
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+
+ def emit_operator_call(self, node, out_c):
+ """Emit operator code at call site."""
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ if func_name == "__nop":
+ return
+
+ if func_name == "__copy":
+ return
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ node_name_upper = node_name.upper()
+ dl_args_name = get_node_args_name(node_name)
+ dl_arg_types_name = get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for arg in node["inputs"]:
+ dl_tensor_name = self._get_node_input_name(arg)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+ assert (
+ dl_tensor_name in self.input_data_
+ ), f"Tensor {dl_tensor_name} not registered ?"
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n")
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+
+ out_c.write(f"#endif \n")
+
+ def emit_params_data(self, target_dir):
+ """Emits the network_data[c,h] files with parameters."""
+ name = self.model_name_
+ name_upper = self.model_name_.upper()
+
+ data_h_name = os.path.join(target_dir, name + "_data.h")
+ data_c_name = os.path.join(target_dir, name + "_data.c")
+
+ out_h = open(data_h_name, "w")
+ out_c = open(data_c_name, "w")
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED(4) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ offset = 0
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ out_h.close()
+ out_c.close()
+
+ def emit_open(self, out_h, out_c):
+ """Emits the network.h file with a few network defines and
+ writes the header part of the network.c file."""
+
+ name = self.model_name_
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ def emit_close(self, out_h, out_c):
+ """Emits the ai_model_info structure."""
+
+ name = self.model_name_
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
Review comment:
since the python emitter is writing code to meet this version, does it makes sense to hardcode the version string here rather than in a header file which may not match the Python at time of compilation?
##########
File path: src/runtime/crt/contrib/stm32/ai_runtime_api.h
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_runtime_api.h
+ * \brief The runtime API for the TVM generated C code.
+ */
+
+#ifndef TVM_RUNTIME_CRT_CONTRIB_STM32_AI_RUNTIME_API_H_
+#define TVM_RUNTIME_CRT_CONTRIB_STM32_AI_RUNTIME_API_H_
+
+#include "ai_platform.h"
+#include "dlpack/dlpack.h" // From TVM
+#include "tvm/runtime/c_runtime_api.h" // From TVM
+
+//
+// This describes current ai_runtime version
+//
+#define AI_PLATFORM_RUNTIME_MAJOR 1
+#define AI_PLATFORM_RUNTIME_MINOR 0
+#define AI_PLATFORM_RUNTIME_MICRO 0
+
+#define AI_STATIC static
+
+#if defined(_MSC_VER)
+#define AI_INLINE __inline
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_INLINE inline
+#elif defined(__CC_ARM)
+#define AI_INLINE __inline
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_INLINE __inline
+#else
+/* Dynamic libraries are not supported by the compiler */
+#endif
+
+/*********************************************************/
+
+typedef uint8_t* ai_ptr;
+
+typedef enum { AI_STATUS_OK = 0, AI_STATUS_ERROR = 1, AI_STATUS_DELEGATE_ERROR = 2 } ai_status;
+
+// =======================================================
+// ai_quantization_info
+//
+// Parameters for asymmetric quantization across a dimension (i.e
+// per output channel quantization).
+// quantized_dimension specifies which dimension the scales and
+// zero_points correspond to.
+// For a particular value in quantized_dimension, quantized values
+// can be converted back to float using:
+// real_value = scale * (quantized_value - zero_point)
+// =======================================================
+
+typedef struct {
+ /*!
+ * \brief The quantization info, if quantized
+ */
+ float* scale;
+ int32_t* zero_point;
+ int32_t dim;
+} ai_quantization_info;
+
+// =======================================================
+// ai_tensor
+// =======================================================
+
+typedef struct {
+ /*!
+ * \brief The TVM tensor.
+ */
+ DLTensor dltensor;
+ /*!
+ * \brief The quantization info, if quantized
+ */
+ ai_quantization_info* quant;
+} ai_tensor;
+
+// =======================================================
+// get_dltensor
+// =======================================================
+AI_STATIC AI_INLINE DLTensor* get_dltensor(ai_tensor* tensor) { return &tensor->dltensor; }
+
+// =======================================================
+// get_tensor_elts
+// =======================================================
+AI_STATIC AI_INLINE uint32_t get_tensor_elts(const ai_tensor* tensor) {
+ const DLTensor* t = &tensor->dltensor;
+ uint32_t elts = 1;
+ for (int i = 0; i < t->ndim; ++i) {
+ elts *= t->shape[i];
+ }
+ return elts;
+}
+
+// =======================================================
+// get_tensor_size
+// =======================================================
+AI_STATIC AI_INLINE uint32_t get_tensor_size(const ai_tensor* tensor) {
+ const DLTensor* t = &tensor->dltensor;
+ uint32_t size = 1;
+ for (int i = 0; i < t->ndim; ++i) {
+ size *= t->shape[i];
+ }
+ size *= (t->dtype.bits * t->dtype.lanes + 7) / 8;
+ return size;
+}
+
+// =======================================================
+// ai_network_info
+// =======================================================
+
+typedef struct {
+ const char* name;
+ const char* datetime;
+ const char* revision;
+ const char* tool_version;
+ const char* api_version;
+ uint16_t n_nodes;
+ uint8_t n_inputs;
+ uint8_t n_outputs;
+ uint32_t activations_size;
+ uint32_t params_size;
+ ai_ptr activations;
+ ai_tensor** inputs;
+ ai_tensor** outputs;
+ const ai_ptr (*ai_get_params)(void);
+ ai_status (*ai_create)(const ai_ptr weights, const ai_ptr activations);
+ ai_status (*ai_destroy)();
+ ai_status (*ai_run)(ai_tensor* input[], ai_tensor* output[]);
+} ai_model_info;
+
+#define AI_MODEL_name(x) (x->name)
+#define AI_MODEL_datetime(x) (x->datetime)
+#define AI_MODEL_revision(x) (x->revision)
+#define AI_MODEL_tool_version(x) (x->tool_version)
+#define AI_MODEL_api_version(x) (x->api_version)
+#define AI_MODEL_n_nodes(x) (x->n_nodes)
+#define AI_MODEL_n_inputs(x) (x->n_inputs)
+#define AI_MODEL_n_outputs(x) (x->n_outputs)
+#define AI_MODEL_activations_size(x) (x->activations_size)
+#define AI_MODEL_params_size(x) (x->params_size)
+#define AI_MODEL_inputs(x) (x->inputs)
+#define AI_MODEL_outputs(x) (x->outputs)
+#define AI_MODEL_activations(x) (x->activations)
+
+// =======================================================
+// Iterator
+//
+// Usage:
+//
+// for (ai_models_iterator it = ai_models_iterator_begin();
+// it != ai_models_iterator_end();
+// it = ai_models_iterator_next(it)) {
+// const char * name = ai_models_iterator_value(it);
+// }
+//
+// =======================================================
+
+typedef uint32_t ai_model_iterator;
+
+ai_model_iterator ai_model_iterator_begin();
+ai_model_iterator ai_model_iterator_next(ai_model_iterator it);
+ai_model_iterator ai_model_iterator_end();
+ai_model_info* ai_model_iterator_value(ai_model_iterator it);
+
+// =======================================================
+// External Interface
+// =======================================================
+
+ai_status ai_create(ai_model_info* nn, void* context, ai_handle* handle);
+
+ai_status ai_destroy(ai_handle handle);
+
+const char* ai_get_error(ai_handle handle);
+
+int32_t ai_get_input_size(ai_handle handle);
+
+int32_t ai_get_output_size(ai_handle handle);
+
+ai_tensor* ai_get_input(ai_handle handle, int32_t index);
+
+ai_tensor* ai_get_output(ai_handle handle, int32_t index);
+
+ai_status ai_run(ai_handle handle);
+
+//
+// Additional methods
+//
+const char* ai_get_name(ai_handle handle);
Review comment:
could you ensure these are tested? in particular i only see usages of name, get_activations, and get_params in the tests
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? ((AI_INTQ_INFO_LIST_FLAGS(AI_BUFFER_META_INFO_INTQ(meta_)) & \
+ AI_BUFFER_META_FLAG_ZEROPOINT_U8) \
+ ? AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_u8, pos_) \
+ : AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_i8, pos_)) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INIT(flags_, intq_info_) \
+ { .flags = (flags_), .intq_info = AI_PACK(intq_info_) }
+
+#define AI_BUFFER_SIZE(buf_) (((buf_)->width) * ((buf_)->height) * ((buf_)->channels))
+
+#define AI_BUFFER_BYTE_SIZE(count_, fmt_) ((((count_)*AI_BUFFER_FMT_GET_BITS(fmt_)) + 4) >> 3)
+
+/*!
+ * @enum buffer formats enum list
+ * @ingroup ai_platform
+ *
+ * List of supported ai_buffer format types.
+ */
+enum {
+ AI_BUFFER_FORMAT_NONE = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_NONE, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_FLOAT = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_FLOAT, 1, 1, 32, 0),
+
+ AI_BUFFER_FORMAT_U8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 0),
+ AI_BUFFER_FORMAT_U16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 0),
+ AI_BUFFER_FORMAT_S8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 0),
+ AI_BUFFER_FORMAT_S16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 0),
+
+ AI_BUFFER_FORMAT_Q = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 0, 0),
+ AI_BUFFER_FORMAT_Q7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 7),
+ AI_BUFFER_FORMAT_Q15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_UQ = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_UQ7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 7),
+ AI_BUFFER_FORMAT_UQ15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_BOOL = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_BOOL, 0, 0, 8, 0),
+};
+
+/******************************************************************************/
+
+/* printf formats */
+#define SSIZET_FMT "%" PRIu32
Review comment:
nit: can't see where these are used
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? ((AI_INTQ_INFO_LIST_FLAGS(AI_BUFFER_META_INFO_INTQ(meta_)) & \
+ AI_BUFFER_META_FLAG_ZEROPOINT_U8) \
+ ? AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_u8, pos_) \
+ : AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_i8, pos_)) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INIT(flags_, intq_info_) \
+ { .flags = (flags_), .intq_info = AI_PACK(intq_info_) }
+
+#define AI_BUFFER_SIZE(buf_) (((buf_)->width) * ((buf_)->height) * ((buf_)->channels))
+
+#define AI_BUFFER_BYTE_SIZE(count_, fmt_) ((((count_)*AI_BUFFER_FMT_GET_BITS(fmt_)) + 4) >> 3)
+
+/*!
+ * @enum buffer formats enum list
+ * @ingroup ai_platform
+ *
+ * List of supported ai_buffer format types.
+ */
+enum {
+ AI_BUFFER_FORMAT_NONE = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_NONE, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_FLOAT = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_FLOAT, 1, 1, 32, 0),
+
+ AI_BUFFER_FORMAT_U8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 0),
+ AI_BUFFER_FORMAT_U16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 0),
+ AI_BUFFER_FORMAT_S8 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 0),
+ AI_BUFFER_FORMAT_S16 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 0),
+
+ AI_BUFFER_FORMAT_Q = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 0, 0),
+ AI_BUFFER_FORMAT_Q7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 8, 7),
+ AI_BUFFER_FORMAT_Q15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 1, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_UQ = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 0, 0),
+ AI_BUFFER_FORMAT_UQ7 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 8, 7),
+ AI_BUFFER_FORMAT_UQ15 = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_Q, 0, 0, 16, 15),
+
+ AI_BUFFER_FORMAT_BOOL = AI_BUFFER_FMT_SET(AI_BUFFER_FMT_TYPE_BOOL, 0, 0, 8, 0),
+};
+
+/******************************************************************************/
+
+/* printf formats */
+#define SSIZET_FMT "%" PRIu32
+#define AII32_FMT "%" PRId32
+#define AIU32_FMT "%" PRIu32
+
+typedef uint8_t ai_custom_type_signature;
+
+typedef void* ai_handle;
+
+typedef void (*ai_handle_func)(void*);
+
+typedef float ai_float;
+typedef double ai_double;
Review comment:
nit: of these i just see ai_float, ai_u8, u16, u32 used
##########
File path: src/runtime/crt/contrib/stm32/ai_runtime_api.h
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_runtime_api.h
+ * \brief The runtime API for the TVM generated C code.
+ */
+
+#ifndef TVM_RUNTIME_CRT_CONTRIB_STM32_AI_RUNTIME_API_H_
+#define TVM_RUNTIME_CRT_CONTRIB_STM32_AI_RUNTIME_API_H_
+
+#include "ai_platform.h"
+#include "dlpack/dlpack.h" // From TVM
+#include "tvm/runtime/c_runtime_api.h" // From TVM
+
+//
+// This describes current ai_runtime version
+//
+#define AI_PLATFORM_RUNTIME_MAJOR 1
+#define AI_PLATFORM_RUNTIME_MINOR 0
+#define AI_PLATFORM_RUNTIME_MICRO 0
+
+#define AI_STATIC static
+
+#if defined(_MSC_VER)
+#define AI_INLINE __inline
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_INLINE inline
+#elif defined(__CC_ARM)
+#define AI_INLINE __inline
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_INLINE __inline
+#else
+/* Dynamic libraries are not supported by the compiler */
+#endif
+
+/*********************************************************/
+
+typedef uint8_t* ai_ptr;
+
+typedef enum { AI_STATUS_OK = 0, AI_STATUS_ERROR = 1, AI_STATUS_DELEGATE_ERROR = 2 } ai_status;
+
+// =======================================================
+// ai_quantization_info
+//
+// Parameters for asymmetric quantization across a dimension (i.e
+// per output channel quantization).
+// quantized_dimension specifies which dimension the scales and
+// zero_points correspond to.
+// For a particular value in quantized_dimension, quantized values
+// can be converted back to float using:
+// real_value = scale * (quantized_value - zero_point)
+// =======================================================
+
+typedef struct {
+ /*!
+ * \brief The quantization info, if quantized
+ */
+ float* scale;
+ int32_t* zero_point;
+ int32_t dim;
+} ai_quantization_info;
+
+// =======================================================
+// ai_tensor
+// =======================================================
+
+typedef struct {
+ /*!
+ * \brief The TVM tensor.
+ */
+ DLTensor dltensor;
+ /*!
+ * \brief The quantization info, if quantized
+ */
+ ai_quantization_info* quant;
+} ai_tensor;
+
+// =======================================================
+// get_dltensor
+// =======================================================
+AI_STATIC AI_INLINE DLTensor* get_dltensor(ai_tensor* tensor) { return &tensor->dltensor; }
Review comment:
i don't see this function exercised--consider deleting it
##########
File path: python/tvm/contrib/stm32/BaseEmitter.py
##########
@@ -0,0 +1,1257 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import tarfile
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """Replace ':' with '_' in names like 'InputImg:0'"""
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def get_tensor_data_name(tensor_name):
+ return tensor_name + ".dltensor.data"
+
+
+def get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+def get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+
+def _emit_tensor_quant(dl_tensor_name, quantization_map, out_c):
+
+ if dl_tensor_name in quantization_map:
+ quantization = quantization_map[dl_tensor_name]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ quant_name = dl_tensor_name + "_quant"
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+
+def emit_tensor_init(dl_tensor_name, tensor, quantization_map, out_c):
+ """Emit tensor definition."""
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ if "device" in tensor:
+ device = tensor["device"]
+ else:
+ device = "kDLCPU"
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ _emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ if quantization_map is not None:
+ quant_name = _emit_tensor_quant(dl_tensor_name, quantization_map, out_c)
+ else:
+ quant_name = None
+
+ #
+ # Contents
+ #
+ out_c.write(f"AI_ALIGNED({8}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n")
Review comment:
a suggestion: there are a lot of "templates" like these that might be easier to understand if you write them as python heredoc and take more advantage of the format() function:
```
AI_TENSOR_TEMPLATE = """\
AI_ALIGNED(8) AI_STATIC ai_tensor {dl_tensor_name} = {{
.dltensor = {{
.data = (ai_ptr)(NULL),
.device = {{ {tensor.get("device", "kDLCPU")} }},
.ndim = {len(tensor["dims"])},
# etc
```
and then
```
out_c.write(AI_TENSOR_TEMPLATE.format(device_name=device_name, tensor=tensor))
```
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,383 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <inttypes.h>
+#include <stddef.h>
+#include <stdint.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, AI_PLATFORM_API_MINOR, AI_PLATFORM_API_MICRO)
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, AI_TOOLS_API_VERSION_MINOR, AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+#define AI_API_ENTRY __declspec(dllexport)
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) AI_CONCAT(AI_ALIGNED_, x)
+#define AI_ALIGNED_1 _Pragma("data_alignment = 1")
+#define AI_ALIGNED_2 _Pragma("data_alignment = 2")
+#define AI_ALIGNED_4 _Pragma("data_alignment = 4")
+#define AI_ALIGNED_8 _Pragma("data_alignment = 8")
+#define AI_ALIGNED_16 _Pragma("data_alignment = 16")
+#define AI_ALIGNED_32 _Pragma("data_alignment = 32")
+#elif defined(__CC_ARM)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_API_ENTRY __attribute__((visibility("default")))
+#define AI_ALIGNED(x) __attribute__((aligned(x)))
+#else
+/* Dynamic libraries are not supported by the compiler */
+#define AI_API_ENTRY /* AI_API_ENTRY */
+#define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#endif
+
+#define AI_HANDLE_PTR(ptr_) ((ai_handle)(ptr_))
+#define AI_HANDLE_NULL AI_HANDLE_PTR(NULL)
+
+#define AI_HANDLE_FUNC_PTR(func) ((ai_handle_func)(func))
+
+#define AI_NETWORK_PARAMS_INIT(params_, activations_) \
+ { .params = params_, .activations = activations_ }
+
+/*! ai_intq_info struct handlers **********************************************/
+#define INTQ_CONST const
+// #define INTQ_CONST
+
+#define AI_INTQ_INFO_LIST(list_) ((list_)->info)
+
+#define AI_INTQ_INFO_LIST_FLAGS(list_) ((list_) ? (list_)->flags : 0)
+
+#define AI_INTQ_INFO_LIST_SIZE(list_) ((list_) ? (list_)->size : 0)
+
+#define AI_HAS_INTQ_INFO_LIST(list_) ((list_) ? (((list_)->info) && ((list_)->size > 0)) : false)
+
+#define AI_INTQ_INFO_LIST_SCALE(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->scale))[(pos_)] \
+ : 0)
+
+#define AI_INTQ_INFO_LIST_ZEROPOINT(list_, type_, pos_) \
+ (((list_) && (list_)->info && ((pos_) < (list_)->size)) \
+ ? ((type_*)((list_)->info->zeropoint))[(pos_)] \
+ : 0)
+
+/*! ai_buffer format handlers *************************************************/
+
+/*!
+ * @enum buffer format definition
+ * @ingroup ai_platform
+ *
+ * 32 bit signed format list.
+ */
+typedef int32_t ai_buffer_format;
+
+/*! ai_buffer_meta flags ******************************************************/
+#define AI_BUFFER_META_HAS_INTQ_INFO (0x1U << 0)
+#define AI_BUFFER_META_FLAG_SCALE_FLOAT (0x1U << 0)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_U8 (0x1U << 1)
+#define AI_BUFFER_META_FLAG_ZEROPOINT_S8 (0x1U << 2)
+
+/*! ai_buffer format variable flags *******************************************/
+#define AI_BUFFER_FMT_TYPE_NONE (0x0)
+#define AI_BUFFER_FMT_TYPE_FLOAT (0x1)
+#define AI_BUFFER_FMT_TYPE_Q (0x2)
+#define AI_BUFFER_FMT_TYPE_BOOL (0x3)
+
+#define AI_BUFFER_FMT_FLAG_CONST (0x1U << 30)
+#define AI_BUFFER_FMT_FLAG_STATIC (0x1U << 29)
+#define AI_BUFFER_FMT_FLAG_IS_IO (0x1U << 27)
+
+#define AI_BUFFER_FMT_PACK(value_, mask_, bits_) (((value_) & (mask_)) << (bits_))
+
+#define AI_BUFFER_FMT_UNPACK(fmt_, mask_, bits_) ((AI_BUFFER_FMT_OBJ(fmt_) >> (bits_)) & (mask_))
+
+#define AI_BUFFER_FMT_OBJ(fmt_) ((ai_buffer_format)(fmt_))
+
+#define AI_BUFFER_FMT_GET_FLOAT(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 24)
+
+#define AI_BUFFER_FMT_GET_SIGN(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x1, 23)
+
+#define AI_BUFFER_FMT_GET_TYPE(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0xF, 17)
+
+#define AI_BUFFER_FMT_GET_BITS(fmt_) AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 7)
+
+#define AI_BUFFER_FMT_SET_BITS(bits_) AI_BUFFER_FMT_PACK((bits_), 0x7F, 7)
+
+#define AI_BUFFER_FMT_GET_FBITS(fmt_) ((ai_i8)AI_BUFFER_FMT_UNPACK(fmt_, 0x7F, 0) - 64)
+
+#define AI_BUFFER_FMT_SET_FBITS(fbits_) AI_BUFFER_FMT_PACK((fbits_) + 64, 0x7F, 0)
+
+#define AI_BUFFER_FMT_SET(type_id_, sign_bit_, float_bit_, bits_, fbits_) \
+ AI_BUFFER_FMT_OBJ(AI_BUFFER_FMT_PACK(float_bit_, 0x1, 24) | \
+ AI_BUFFER_FMT_PACK(sign_bit_, 0x1, 23) | AI_BUFFER_FMT_PACK(0, 0x3, 21) | \
+ AI_BUFFER_FMT_PACK(type_id_, 0xF, 17) | AI_BUFFER_FMT_PACK(0, 0x7, 14) | \
+ AI_BUFFER_FMT_SET_BITS(bits_) | AI_BUFFER_FMT_SET_FBITS(fbits_))
+
+#define AI_BUFFER_FMT_GET(fmt_) (AI_BUFFER_FMT_OBJ(fmt_) & 0x01FFFFFF)
+
+#define AI_BUFFER_FORMAT(buf_) AI_BUFFER_FMT_GET((buf_)->format)
+#define AI_BUFFER_WIDTH(buf_) ((buf_)->width)
+#define AI_BUFFER_HEIGHT(buf_) ((buf_)->height)
+#define AI_BUFFER_CHANNELS(buf_) ((buf_)->channels)
+#define AI_BUFFER_N_BATCHES(buf_) ((buf_)->n_batches)
+#define AI_BUFFER_DATA(buf_, type_) ((type_*)((buf_)->data))
+
+#define AI_BUFFER_META_INFO(buf_) ((buf_)->meta_info)
+
+#define AI_BUFFER_META_INFO_INTQ(meta_) \
+ ((meta_) && ((meta_)->flags & AI_BUFFER_META_HAS_INTQ_INFO)) ? ((meta_)->intq_info) \
+ : NULL // NOLINT
+
+#define AI_BUFFER_META_INFO_INTQ_GET_SCALE(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? AI_INTQ_INFO_LIST_SCALE(AI_BUFFER_META_INFO_INTQ(meta_), ai_float, pos_) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INTQ_GET_ZEROPOINT(meta_, pos_) \
+ ((AI_BUFFER_META_INFO_INTQ(meta_)) \
+ ? ((AI_INTQ_INFO_LIST_FLAGS(AI_BUFFER_META_INFO_INTQ(meta_)) & \
+ AI_BUFFER_META_FLAG_ZEROPOINT_U8) \
+ ? AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_u8, pos_) \
+ : AI_INTQ_INFO_LIST_ZEROPOINT(AI_BUFFER_META_INFO_INTQ(meta_), ai_i8, pos_)) \
+ : 0)
+
+#define AI_BUFFER_META_INFO_INIT(flags_, intq_info_) \
Review comment:
nit: can't see where this is used
##########
File path: src/runtime/crt/contrib/stm32/ai_runtime_api.h
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_runtime_api.h
+ * \brief The runtime API for the TVM generated C code.
+ */
+
+#ifndef TVM_RUNTIME_CRT_CONTRIB_STM32_AI_RUNTIME_API_H_
+#define TVM_RUNTIME_CRT_CONTRIB_STM32_AI_RUNTIME_API_H_
+
+#include "ai_platform.h"
+#include "dlpack/dlpack.h" // From TVM
+#include "tvm/runtime/c_runtime_api.h" // From TVM
+
+//
+// This describes current ai_runtime version
+//
+#define AI_PLATFORM_RUNTIME_MAJOR 1
+#define AI_PLATFORM_RUNTIME_MINOR 0
+#define AI_PLATFORM_RUNTIME_MICRO 0
+
+#define AI_STATIC static
+
+#if defined(_MSC_VER)
+#define AI_INLINE __inline
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
+#define AI_INLINE inline
+#elif defined(__CC_ARM)
+#define AI_INLINE __inline
+/* Keil disallows anonymous union initialization by default */
+#pragma anon_unions
+#elif defined(__GNUC__)
+#define AI_INLINE __inline
+#else
+/* Dynamic libraries are not supported by the compiler */
+#endif
+
+/*********************************************************/
+
+typedef uint8_t* ai_ptr;
+
+typedef enum { AI_STATUS_OK = 0, AI_STATUS_ERROR = 1, AI_STATUS_DELEGATE_ERROR = 2 } ai_status;
+
+// =======================================================
+// ai_quantization_info
+//
+// Parameters for asymmetric quantization across a dimension (i.e
+// per output channel quantization).
+// quantized_dimension specifies which dimension the scales and
+// zero_points correspond to.
+// For a particular value in quantized_dimension, quantized values
+// can be converted back to float using:
+// real_value = scale * (quantized_value - zero_point)
+// =======================================================
+
+typedef struct {
+ /*!
+ * \brief The quantization info, if quantized
+ */
+ float* scale;
+ int32_t* zero_point;
+ int32_t dim;
+} ai_quantization_info;
+
+// =======================================================
+// ai_tensor
+// =======================================================
+
+typedef struct {
+ /*!
+ * \brief The TVM tensor.
+ */
+ DLTensor dltensor;
+ /*!
+ * \brief The quantization info, if quantized
+ */
+ ai_quantization_info* quant;
+} ai_tensor;
+
+// =======================================================
+// get_dltensor
+// =======================================================
+AI_STATIC AI_INLINE DLTensor* get_dltensor(ai_tensor* tensor) { return &tensor->dltensor; }
+
+// =======================================================
+// get_tensor_elts
+// =======================================================
+AI_STATIC AI_INLINE uint32_t get_tensor_elts(const ai_tensor* tensor) {
+ const DLTensor* t = &tensor->dltensor;
+ uint32_t elts = 1;
+ for (int i = 0; i < t->ndim; ++i) {
+ elts *= t->shape[i];
+ }
+ return elts;
+}
+
+// =======================================================
+// get_tensor_size
+// =======================================================
+AI_STATIC AI_INLINE uint32_t get_tensor_size(const ai_tensor* tensor) {
+ const DLTensor* t = &tensor->dltensor;
+ uint32_t size = 1;
+ for (int i = 0; i < t->ndim; ++i) {
+ size *= t->shape[i];
+ }
+ size *= (t->dtype.bits * t->dtype.lanes + 7) / 8;
+ return size;
+}
+
+// =======================================================
+// ai_network_info
+// =======================================================
+
+typedef struct {
+ const char* name;
+ const char* datetime;
+ const char* revision;
+ const char* tool_version;
+ const char* api_version;
+ uint16_t n_nodes;
+ uint8_t n_inputs;
+ uint8_t n_outputs;
+ uint32_t activations_size;
+ uint32_t params_size;
+ ai_ptr activations;
+ ai_tensor** inputs;
+ ai_tensor** outputs;
+ const ai_ptr (*ai_get_params)(void);
+ ai_status (*ai_create)(const ai_ptr weights, const ai_ptr activations);
+ ai_status (*ai_destroy)();
+ ai_status (*ai_run)(ai_tensor* input[], ai_tensor* output[]);
+} ai_model_info;
+
+#define AI_MODEL_name(x) (x->name)
+#define AI_MODEL_datetime(x) (x->datetime)
+#define AI_MODEL_revision(x) (x->revision)
+#define AI_MODEL_tool_version(x) (x->tool_version)
+#define AI_MODEL_api_version(x) (x->api_version)
+#define AI_MODEL_n_nodes(x) (x->n_nodes)
+#define AI_MODEL_n_inputs(x) (x->n_inputs)
+#define AI_MODEL_n_outputs(x) (x->n_outputs)
+#define AI_MODEL_activations_size(x) (x->activations_size)
+#define AI_MODEL_params_size(x) (x->params_size)
+#define AI_MODEL_inputs(x) (x->inputs)
+#define AI_MODEL_outputs(x) (x->outputs)
+#define AI_MODEL_activations(x) (x->activations)
+
+// =======================================================
+// Iterator
Review comment:
could you add a test to exercise these functions?
##########
File path: python/tvm/contrib/stm32/BaseEmitter.py
##########
@@ -0,0 +1,1257 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import tarfile
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """Replace ':' with '_' in names like 'InputImg:0'"""
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def get_tensor_data_name(tensor_name):
+ return tensor_name + ".dltensor.data"
+
+
+def get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
Review comment:
suggest to use a python dict for this:
```
C_TYPE_BY_DLTYPE = {
"uint64": "kDLUinit, 64, 1",
# ...
}
```
and
```
try:
return C_TYPE_BY_DLTYPE[dltype]
except KeyError:
raise ValueError(f"Data type {dltype} is not supported")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-873293622
@stoa i posted up a proposal RFC with the goal to clarify the split between c_runtime_api and c_backend_api: https://discuss.tvm.apache.org/t/pre-rfc-api-change-formalizing-c-backend-api/10380/2
hopefully this helps to clarify what is required from TVM from the generated model operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-809509753
Hello, Andrew
I just have posted this: https://discuss.tvm.apache.org/t/rfc-standalone-code-generation-and-c-runtime-for-stm32-bare-metal-devices/9562
Please, feel free to correct if I have missed something from on-going projects. The RFC is a basis to see if/how we can contribute on the micro TVM topics. I have tried to describe our prospective.
Looking forward to the discussion.
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r616854241
##########
File path: apps/stm32/Makefile
##########
@@ -0,0 +1,90 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Top level Makefile for the STM32 microcontroller project.
+
+#
+# Set the path to your ARM Toolset.
+# Only gcc is supported at this time.
+#
+#export ARM_PATH ?=
+ifndef ARM_PATH
+$(error ARM_PATH must be set and point at your GCC ARM compiler installation)
+endif
+
+#
+# Set the path to your TVM compiler installation
+#
+#export TVM_PATH ?=
+ifndef TVM_PATH
+$(error TVM_PATH must be set and point at your TVM stack installation)
+endif
+
+#
+# Set the path to your CubeMX installation
+#
+#export X_CUBE_PATH ?=
+ifndef X_CUBE_PATH
+$(error X_CUBE_PATH must be set and point at your STM32 Cube Firmware installation)
+endif
+
+#export X_CUBE_TOOL_PATH ?=
+ifndef X_CUBE_TOOL_PATH
+$(error X_CUBE_TOOL_PATH must be set and point at your STM32 Cube Programmer installation)
+endif
+
+#
+# Setup path to your network generated files
+#
+#export MODEL_PATH =
+ifndef MODEL_PATH
+$(error MODEL_PATH must be set and point at your model implementation generated with the TVM compiler)
+endif
+
+BOARD ?= h747idisco
Review comment:
This is a board with a lot of memory - so that we can run pretty much any model without spending too much time on making the model fit into the memory limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r713991503
##########
File path: src/runtime/crt/contrib/stm32/runtime.c
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file runtime.c
+ * \brief A minimal "C" runtime support required by the TVM
+ * generated C code. Declared in "runtime/c_backend_api.h"
+ * and "runtime/c_runtime_api.h"
+ */
+
+#include <assert.h>
+#include <malloc.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <tvm/runtime/c_backend_api.h>
+#include <tvm/runtime/crt/error_codes.h>
+
+static char* g_last_error = NULL;
+
+// ====================================================
+// TVMPlatformMemoryAllocate
+// ====================================================
+tvm_crt_error_t TVMPlatformMemoryAllocate(size_t num_bytes, DLDevice dev, void** out_ptr) {
Review comment:
Yes. We will provide timing and monitor functions eventually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-879899854
@areusch
Hello, Andrew
I have posted a message on discord. Not sure I have used it properly. Let me know if you've received.
cheers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r616871415
##########
File path: apps/stm32/.clang-format
##########
@@ -0,0 +1,2 @@
+DisableFormat: true
Review comment:
It only concerns the project specific files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-875270854
@stoa it might help for us to have a higher-bandwidth discussion on this topic (we can place the results in this thread). if you want to contact me on the [discord](https://tvm.apache.org/community) we may be able to come to a plan more quickly.
initially my thoughts were that, since this PR was contributing a new implementation of c_backend_api and the API surface was considerably different than what is in `src/runtime/micro`, there were enough pieces similar to `src/runtime/crt` that it made sense to place it with the CRT. however, as we've been aligning on this PR, it now seems like we've been able to accommodate any platform differences for STM32 using the standard TVM C runtime plus the TVMPlatform APIs. i think this PR now has these pieces:
1. a code generator to produce the STM32-facing API from a Model Library Format artifact
2. implementation of the TVMPlatform APIs
3. implementations of the types used by STM32 API
4. unit tests for the above
So now it seems like pieces 1-3 more naturally belong alongside those for the other microTVM platforms (Zephyr and Arduino (soon)). With the [Project API PR](https://github.com/apache/tvm/pull/8380), it's proposed to move that code to `apps/microtvm/<platform>`. I'm happy to consider other proposals for where it should live, too. Or, if you plan to contribute more pieces later on that might change the picture a bit, we could consider another home for this code.
I did want note that although support for these platforms is compartmentalized outside of `src` in the TVM codebase, we still test them as much as is feasible in the TVM CI so that we can consider them first-class integrations of TVM. There is precedent in TVM for doing this (e.g. `apps/bundle_deploy`, which is the recommended way to build a static POSIX binary to deploy TVM code), so I would think that the CI should protect them from bitrot.
Another benefit is that it places all of the stm-specific code in a single location. this makes it easier for reviewers to tell when to CC you guys on PRs that originate from the community but which modify the STM32 code. TVM doesn't have a codified [CODEOWNERS](https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-repository-on-github/about-code-owners) but it still nonetheless helps to have clearly-delineated code.
Let me know what you think--it'd be great to align on this last piece!
-Andrew
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-823535548
@areusch
It would be difficult to use the STM32 without the need to download any package from our site - you need at least the BSP, and the HAL helps a lot. This would be the minimum to be reasonable.
We plan to supply the X-Cube with TVM integrated with it. It is easier to generate a project using the X-Cube. Once the project with the board configuration is generated, we can work with it outside of the Cube, if desired. For example, I usually do this - I am traditionally a command line person.
The apps/stm32 demo application is a X-Cube generated project (board configuration) where we added our application for Performance Measurement and Validation of ML models. The models are compiled by TVM. I thought it is a good idea to have such a demo bundled with the TVM source base, but you are right - there is still a need to download some basic board support packages. However, notice that this demo application should not add to the CI complexity since it is not part of the CI, just an application note if you wish.
If you believe it is better - we can set this application up separately from TVM. Perhaps, there are other applications that are setup in this way ? We would follow their guideline then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r638263670
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
Review comment:
please alphabetize the system imports
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
Review comment:
it could be easier to understand to write as `DBAR = "=" * 60
##########
File path: src/runtime/crt/contrib/stm32/.clang-format
##########
@@ -0,0 +1,2 @@
+DisableFormat: true
Review comment:
for code underneath `src`, i think we'd like to avoid any special `.clang-format` directives. in `apps`, we may write code meant to compile with other frameworks, but code in the `src` tree is for TVM core code, and I don't think we should allow for deviation from the project's code style guide here.
cc @tqchen
##########
File path: src/runtime/crt/contrib/stm32/stm32lib.h
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file stm32lib.h
+ * \brief Support for the TVM code generation for STM32 ARM targets.
+ */
+
+// LINT_C_FILE
+
+#ifndef TVM_RUNTIME_CRT_CONTRIB_STM32_STM32LIB_H_
+#define TVM_RUNTIME_CRT_CONTRIB_STM32_STM32LIB_H_
+
+#include <stdint.h>
+
+#ifdef __arm__
+#include "cmsis_compiler.h"
+#endif
+
+// =========================================================
+// conv2d_nhwc_int8_smlad_reset_
+// =========================================================
+__attribute__((always_inline)) inline static int32_t
+conv2d_nhwc_int8_smlad_reset_(
+ int32_t * conv
+) {
+ *conv = 0;
+ return 0;
+}
+
+// =========================================================
+// conv2d_nhwc_int8_smlad_update_
+// =========================================================
+__attribute__((always_inline)) inline static int32_t
+conv2d_nhwc_int8_smlad_update_(
+ int8_t * input,
+ int8_t * weights,
+ int32_t * conv,
+ int16_t omaps,
+ int16_t channels
+) {
+ // te.sum((a[k]*b[k,i]).astype('int32'), axis=k
+
+#ifdef __arm__
+ //
+ // Load and extend int8x2 => int16x2
+ //
+ int32_t in = *(int32_t*)input; // TODO(stoa): fix it !!
+ //
+ // Pack 2 kernel values
+ //
+ int8_t ker0 = *(weights+0*channels);
+ int8_t ker1 = *(weights+1*channels);
+ int32_t ker = ((ker1<<16)&0x00FF)|(ker0&0x00FF);
+
+ *(conv) = __SMLAD(in, ker, *(conv));
+#else
+ int8_t in0 = *(input+0);
+ int8_t ker0 = *(weights+0*channels);
+ int8_t in1 = *(input+1);
+ int8_t ker1 = *(weights+1*channels);
+ *(conv) = *(conv) + in0*ker0 + in1*ker1;
+#endif
+
+ return 0;
+}
+
+#if 0
Review comment:
remove
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_create ( \n")
+ out_h.write(f" const ai_ptr weights, \n")
+ out_h.write(f" const ai_ptr activations \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_destroy (); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_create \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_create( \n")
+ out_c.write(f" const ai_ptr weights, \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" ai_status status = AI_STATUS_OK;\n")
+ out_c.write(f" status = {name}_configure_weights (weights); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" status = {name}_configure_activations (activations); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_destroy \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_destroy () \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+
+ # ==================================================================
+ # emit_code
+ # ==================================================================
+ def emit_code(self, dest_dir, model_name):
+ """ Emit the C code implementing the model. """
+
+ #
+ # Build the directory structure
+ #
+ if os.path.exists(dest_dir):
Review comment:
it seems like this behavior would be pretty unexpected to a user if there was e.g.a. typo. suggest just erroring if the dir exists.
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
Review comment:
suggest an extended docstring, like:
```
out_h.write(textwrap.dedent("""\
AI_API_ENTRY
ai_status ai_{name}_create (
const ai_ptr weights,
...
"""))
##########
File path: src/runtime/crt/contrib/stm32/ai_platform.h
##########
@@ -0,0 +1,421 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_platform.h
+ * \brief Definitions of AI platform public APIs types
+ */
+
+// LINT_C_FILE
+
+#ifndef AI_PLATFORM_H
+#define AI_PLATFORM_H
+#pragma once
+
+#include <stdint.h>
+#include <stddef.h>
+#include <inttypes.h>
+
+#ifndef AI_PLATFORM_API_MAJOR
+#define AI_PLATFORM_API_MAJOR (0)
+#endif
+#ifndef AI_PLATFORM_API_MINOR
+#define AI_PLATFORM_API_MINOR (6)
+#endif
+#ifndef AI_PLATFORM_API_MICRO
+#define AI_PLATFORM_API_MICRO (0)
+#endif
+
+#define AI_PLATFORM_API_VERSION \
+ AI_VERSION(AI_PLATFORM_API_MAJOR, \
+ AI_PLATFORM_API_MINOR, \
+ AI_PLATFORM_API_MICRO)
+
+
+#ifndef AI_TOOLS_API_VERSION_MAJOR
+#define AI_TOOLS_API_VERSION_MAJOR (1)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MINOR
+#define AI_TOOLS_API_VERSION_MINOR (4)
+#endif
+#ifndef AI_TOOLS_API_VERSION_MICRO
+#define AI_TOOLS_API_VERSION_MICRO (0)
+#endif
+
+#define AI_TOOLS_API_VERSION \
+ AI_VERSION(AI_TOOLS_API_VERSION_MAJOR, \
+ AI_TOOLS_API_VERSION_MINOR, \
+ AI_TOOLS_API_VERSION_MICRO)
+
+#define AI_TOOLS_API_VERSION_1_3 \
+ AI_VERSION(1, 3, 0)
+
+/******************************************************************************/
+#ifdef __cplusplus
+#define AI_API_DECLARE_BEGIN extern "C" {
+#define AI_API_DECLARE_END }
+#else
+#include <stdbool.h>
+#define AI_API_DECLARE_BEGIN /* AI_API_DECLARE_BEGIN */
+#define AI_API_DECLARE_END /* AI_API_DECLARE_END */
+#endif
+
+/******************************************************************************/
+#if defined(_MSC_VER)
+ #define AI_API_ENTRY __declspec(dllexport)
+ #define AI_ALIGNED(x) /* AI_ALIGNED(x) */
+#elif defined(__ICCARM__) || defined(__IAR_SYSTEMS_ICC__)
Review comment:
thinking again about the SDK - TVM split, I see a bunch of stuff here that I think makes sense in TVM, e.g. `struct ai_network_report_` and dependencies. then I see some other stuff e.g. this ICC-specific logic here. It seems pretty hard to test this--wondering if it's necessary to place all of this in `ai_platform.h`?
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
Review comment:
remove commented code
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
Review comment:
```suggestion
def _get_num_tensor_elements(shape):
size = 1
for dim in shape:
size = size * dim
return size
```
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
Review comment:
suggest specifying units of size: `_get_tensor_size_bytes`
##########
File path: src/runtime/crt/contrib/stm32/runtime.c
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file runtime.c
+ * \brief A minimal "C" runtime support required by the TVM
+ * generated C code. Declared in "runtime/c_backend_api.h"
+ * and "runtime/c_runtime_api.h"
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <string.h>
+#include <malloc.h>
+
+#define _USE_TVM_BACKEND_API_
+
+#ifdef _USE_TVM_BACKEND_API_
+#include <tvm/runtime/c_backend_api.h>
+#include <tvm/runtime/crt/error_codes.h>
+#else
+#include "tvm/runtime/c_runtime_api.h"
+#endif
+
+static char * g_last_error = NULL;
+
+#ifdef _USE_TVM_BACKEND_API_
+
+// ====================================================
+// TVMPlatformMemoryAllocate
+// ====================================================
+tvm_crt_error_t TVMPlatformMemoryAllocate(size_t num_bytes, DLDevice dev, void** out_ptr) {
+#ifdef __arm__
+ *out_ptr = malloc(num_bytes);
+#else // _x86_
+ *out_ptr = malloc(num_bytes);
+#endif
+ return (*out_ptr == NULL) ? kTvmErrorPlatformNoMemory : kTvmErrorNoError;
+}
+
+// ====================================================
+// TVMPlatformMemoryFree
+// ====================================================
+tvm_crt_error_t TVMPlatformMemoryFree(void* ptr, DLDevice dev) {
+ free(ptr);
+ return kTvmErrorNoError;
+}
+
+// ====================================================
+// TVMFuncRegisterGlobal
+// ====================================================
+int TVMFuncRegisterGlobal(const char* name, TVMFunctionHandle f, int override) {
+ return 0;
Review comment:
if unimplemented, return an error code
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
Review comment:
where no docstring is provided, style for this project would have you omit the dbar + name e.g. suggest to delete this part here and elsewhere:
```
# ==========================================================
# get_input_tensor_name
# ==========================================================
```
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
Review comment:
suggest to break lines to make it clearer
```suggestion
dst = (
"#include <stdio.h>\n"
"#include <math.h>\n"
'#include "stm32lib.h"\n\n' ""
)
```
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
Review comment:
suggest adding a unit suffix: `data_alignment_bytes`. also for Python constants, suggest to move this to class level:
```
class CodeEmitter:
DATA_ALIGNMENT_BYTES = 8
```
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
Review comment:
i think you could just write this as `return dst + src`, no? curious why you need to splitlines() here
##########
File path: src/runtime/crt/contrib/stm32/runtime.c
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file runtime.c
+ * \brief A minimal "C" runtime support required by the TVM
+ * generated C code. Declared in "runtime/c_backend_api.h"
+ * and "runtime/c_runtime_api.h"
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <string.h>
+#include <malloc.h>
+
+#define _USE_TVM_BACKEND_API_
+
+#ifdef _USE_TVM_BACKEND_API_
+#include <tvm/runtime/c_backend_api.h>
+#include <tvm/runtime/crt/error_codes.h>
+#else
+#include "tvm/runtime/c_runtime_api.h"
+#endif
+
+static char * g_last_error = NULL;
+
+#ifdef _USE_TVM_BACKEND_API_
+
+// ====================================================
+// TVMPlatformMemoryAllocate
+// ====================================================
+tvm_crt_error_t TVMPlatformMemoryAllocate(size_t num_bytes, DLDevice dev, void** out_ptr) {
+#ifdef __arm__
+ *out_ptr = malloc(num_bytes);
+#else // _x86_
+ *out_ptr = malloc(num_bytes);
+#endif
+ return (*out_ptr == NULL) ? kTvmErrorPlatformNoMemory : kTvmErrorNoError;
+}
+
+// ====================================================
+// TVMPlatformMemoryFree
+// ====================================================
+tvm_crt_error_t TVMPlatformMemoryFree(void* ptr, DLDevice dev) {
+ free(ptr);
+ return kTvmErrorNoError;
+}
+
+// ====================================================
+// TVMFuncRegisterGlobal
+// ====================================================
+int TVMFuncRegisterGlobal(const char* name, TVMFunctionHandle f, int override) {
+ return 0;
+}
+
+// ====================================================
+// TVMPlatformAbort
+// ====================================================
+void __attribute__((noreturn)) TVMPlatformAbort(tvm_crt_error_t code) {
Review comment:
since it is marked noreturn, should at least spin in a for(;;) ;
##########
File path: src/runtime/crt/contrib/stm32/ai_runtime_api.c
##########
@@ -0,0 +1,410 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_runtime_api.c
+ * \brief The runtime API for the TVM generated C code.
+ */
+
+// LINT_C_FILE
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+
+#include "ai_runtime_api.h"
+
+// =======================================================
+// ai_network_t
+// =======================================================
+
+typedef struct {
+ ai_model_info * info;
+ ai_tensor ** inputs;
+ ai_tensor ** outputs;
+ ai_ptr activations;
+ const char * error;
+} ai_network_t;
+
+//
+// .nn_models_info section
+//
+extern uint32_t __models_section_start__;
+extern uint32_t __models_section_end__;
+
+uint32_t _modelsSection_start = (uint32_t)(&__models_section_start__);
+uint32_t _modelsSection_end = (uint32_t)(&__models_section_end__);
+
+// =======================================================
+// Iterator
+// =======================================================
+ai_model_iterator ai_model_iterator_begin() {
+ return _modelsSection_start; // begin()
+}
+
+ai_model_iterator ai_model_iterator_end() {
+ return _modelsSection_end;
+}
+
+ai_model_iterator ai_model_iterator_next(ai_model_iterator idx) {
+ return (idx+sizeof(ai_model_info));
+}
+
+ai_model_info * ai_model_iterator_value(ai_model_iterator idx) {
+ return (ai_model_info*)idx;
+}
+
+#if 0
Review comment:
remove such code, here and elsewhere
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
Review comment:
remove
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
Review comment:
for documentation strings, please follow [numpydoc style](https://tvm.apache.org/docs/contribute/code_guide.html#python-code-styles).
this will allow our sphinx documentation generator to build [API docs](https://tvm.apache.org/docs/api/python/index.html) for the code.
e.g.
```
def _fix_name(node_name):
"""Replace ':' with '_' in names like 'InputImg:0'."""
```
##########
File path: src/runtime/crt/contrib/stm32/stm32lib.h
##########
@@ -0,0 +1,198 @@
+/*
Review comment:
could you explain this file? generally speaking, TVM operator implementations need to live in `python/tvm/topi`. we do have some facility to link to external operator implementations via `tir.call_extern` nodes.
##########
File path: src/runtime/crt/contrib/stm32/crt_config.h
##########
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file tvm/runtime/crt_config.h.template
+ * \brief Template for CRT configuration, to be modified on each target.
+ */
+#ifndef TVM_RUNTIME_CRT_CONTRIB_STM32_CRT_CONFIG_H_
+#define TVM_RUNTIME_CRT_CONTRIB_STM32_CRT_CONFIG_H_
+
+#include <tvm/runtime/crt/logging.h>
+
+/*! Log level of the CRT runtime */
+#define TVM_CRT_LOG_LEVEL TVM_CRT_LOG_LEVEL_DEBUG
Review comment:
i think we need to include the full `crt_config.h` if we're including code from `<tvm/runtime/crt>`
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -0,0 +1,421 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import struct
+
+#
+# Disable GPU usage information:
+#
+#
+# 0 = all messages are logged (default behavior)
+# 1 = INFO messages are not printed
+# 2 = INFO and WARNING messages are not printed
+# 3 = INFO, WARNING, and ERROR messages are not printed
+os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
Review comment:
suggest to allow the OS environment to override this.
```suggestion
if "TF_CPP_MIN_LOG_LEVEL" not in os.environ:
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
```
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -0,0 +1,421 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import struct
+
+#
+# Disable GPU usage information:
+#
+#
+# 0 = all messages are logged (default behavior)
+# 1 = INFO messages are not printed
+# 2 = INFO and WARNING messages are not printed
+# 3 = INFO, WARNING, and ERROR messages are not printed
+os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
+
+import numpy as np
+
+import tensorflow as tf
+
+import tvm
+import tvm.relay as relay
+from tvm.contrib import stm32
+from tvm import testing
+
+TEST_IMAGES = [
+ "00.raw",
+ "01.raw",
+ "02.raw",
+ "03.raw",
+ "04.raw",
+ "05.raw",
+ "06.raw",
+ "07.raw",
+ "08.raw",
+ "09.raw",
+]
+
+BUILD_DIR = "build"
+
+# ==================================================================
+# dump_image
+# ==================================================================
+def dump_image(filename, image):
+ # Flatten image
+ image_data = image.flatten()
+ outputRaw = []
+ # Raw binary format
+ for i in range(0, len(image_data)):
+ outputRaw.append(struct.pack("<B", int(image_data[i]) & 0xFF))
+
+ # Dump image in raw binary format
+ f = open(filename, "wb")
+ for i in range(0, len(outputRaw)):
+ f.write(outputRaw[i])
+ f.close()
+
+
+# ==================================================================
+# scale_input_data
+# ==================================================================
+def scale_input_data(input_details, data):
+ if input_details["dtype"] == np.uint8 or input_details["dtype"] == np.int8:
+ input_scale, input_zero_point = input_details["quantization"]
+ print(
+ "== TFLite input quantization: scale={}, zero={}".format(input_scale, input_zero_point)
+ )
+ data = data / input_scale + input_zero_point
+ data = data.astype(input_details["dtype"])
+ return data
+
+
+# ==================================================================
+# scale_output_data
+# ==================================================================
+def scale_output_data(output_details, data):
+ if output_details["dtype"] == np.uint8 or output_details["dtype"] == np.int8:
+ output_scale, output_zero_point = output_details["quantization"]
+ print(
+ "== TFLite output quantization: scale={}, zero={}".format(
+ output_scale, output_zero_point
+ )
+ )
+ data = data.astype(np.float32)
+ data = (data - output_zero_point) * output_scale
+ return data
+
+
+# ========================================================
+# get_tflite_model
+# ========================================================
+def get_tflite_model(model_path):
+
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()
+ output_details = interpreter.get_output_details()
+
+ #
+ # Figure out shapes and
+ #
+
+ shape_dict = {}
+ dtype_dict = {}
+
+ for input in input_details:
+ input_name = input["name"]
+ input_shape = input["shape"].tolist()
+ input_dtype = str(np.dtype(input["dtype"]))
+ shape_dict[input_name] = input_shape
+ dtype_dict[input_name] = input_dtype
+
+ #
+ # Save the model
+ #
+
+ #
+ # Load the TFLite Model for TVM:
+ #
+ # https://docs.tvm.ai/tutorials/frontend/from_tflite.html
+ # https://jackwish.net/tflite/docs/
+
+ model_buf = open(model_path, "rb").read()
+
+ #
+ # Get TFLite model from buffer
+ #
+ try:
+ import tflite
+
+ model = tflite.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model)
+ except AttributeError:
+ import tflite.Model
+
+ model = tflite.Model.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model.Model)
+
+ print("TVM: Importing a TFLite model ...")
+
+ return model, shape_dict, dtype_dict
+
+
+# ========================================================
+# extract_tflite_quantization
+# ========================================================
+
+
+def _make_qnn_params(quantization):
+ qnn_params = {}
+ qnn_params["min"] = quantization.MinAsNumpy()
+ qnn_params["max"] = quantization.MaxAsNumpy()
+ qnn_params["scale"] = quantization.ScaleAsNumpy()
+ qnn_params["zero_point"] = quantization.ZeroPointAsNumpy()
+ qnn_params["dim"] = quantization.QuantizedDimension()
+ # print(" Quantization: ({}, {}), s={}, z={}, dim={}".format(min, max, scale, zero_point, dim))
+ return qnn_params
+
+
+def extract_tflite_quantization(model):
+
+ assert model.SubgraphsLength() == 1, "only support one subgraph (main subgraph)"
+
+ subgraph = model.Subgraphs(0)
+
+ quantization_info = {}
+
+ # model inputs / outputs
+ model_inputs = subgraph.InputsAsNumpy()
+ model_outputs = subgraph.OutputsAsNumpy()
+
+ for node_id in model_inputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ # print("== Input[{}]: {} shape={} type={}".format(node_id, tensor_name, tensor.ShapeAsNumpy(), tensor_type))
+ dl_tensor_name = stm32.get_input_tensor_name(tensor_name)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ for node_id in model_outputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ # print("== Output[{}]: {} shape={} type={}".format(node_id, tensor_name, tensor.ShapeAsNumpy(), tensor_type))
+ #
+ # TODO: TVM does not preserve the output tensor names.
+ # Eventually, we should be able to form a valid name.
+ #
+ dl_tensor_name = stm32.get_output_tensor_name(tensor_name, 0)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ return quantization_info
+
+
+# ========================================================
+# run_tflite_model
+# ========================================================
+def run_tflite_model(model_path, image_path):
+
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()[0]
+ output_details = interpreter.get_output_details()[0]
+
+ #
+ # Run test images
+ #
+ tf_results = np.empty(shape=[len(TEST_IMAGES), 10], dtype=np.float)
+ for i, filename in enumerate(TEST_IMAGES):
+ image_data = np.fromfile(os.path.join(image_path, filename), dtype="uint8")
+ #
+ # Run the TFLite model: channels last
+ #
+ image_data = image_data.reshape([1, 28, 28, 1])
+ #
+ # Normalize the input data
+ #
+ image_data = image_data / 255.0
+
+ image_data = scale_input_data(input_details, image_data)
+ interpreter.set_tensor(input_details["index"], image_data)
+ interpreter.invoke()
+ tf_results[i] = interpreter.get_tensor(output_details["index"])
+ tf_results[i] = scale_output_data(output_details, tf_results[i])
+
+ print(f"== [{i}] TFLite Output:")
+ print(tf_results[i])
+
+ return tf_results
+
+
+# ========================================================
+# run_tvm_model
+# ========================================================
+def run_tvm_model(build_dir, model_name, target_dir, image_path):
+
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+
+ tvm_results_name = os.path.join(build_dir, "tvm_results.txt")
+
+ #
+ # Build the model
+ #
+ tvm_dir = os.path.join(curr_path, "..", "..", "..")
+ test_dir = os.path.join(tvm_dir, "tests", "crt", "contrib", "stm32")
+
+ command = f"make -f {test_dir}/Makefile TVM_PATH={tvm_dir} MODEL_PATH={target_dir} BUILD_PATH={build_dir} IMAGE_PATH={image_path}"
+ print(f"{command}")
+ os.system(command)
+ #
+ # Run
+ #
+ command = f"{target_dir}/{model_name}.exe"
+ print(f"{command}")
+ os.system(command)
+
+ tvm_results = np.loadtxt(tvm_results_name)
+ print(f"== TVM Output:\n {tvm_results}")
+
+ #
+ # Clean temporary image files
+ #
+ if os.path.exists(tvm_results_name):
+ os.remove(tvm_results_name)
+
+ return tvm_results
+
+
+# ========================================================
+# check_network
+# ========================================================
+def check_network(build_dir, target_name, model_path, image_path):
+
+ model_name = "network"
+
+ target_dir = os.path.join(build_dir, target_name + "_gen")
+
+ model, shape_dict, dtype_dict = get_tflite_model(model_path)
+ mod, params = relay.frontend.from_tflite(model, shape_dict, dtype_dict)
+
+ #
+ # Build a TVM C module for the ARM CPU (without compiling the kernels
+ # library to the object code form):
+ #
+ target = "c -device=arm_cpu"
+ opt_level = 3
Review comment:
since this isn't used elsewhere in this file, suggest to just inline on line 323
##########
File path: tests/lint/check_file_type.py
##########
@@ -142,6 +142,35 @@
"apps/microtvm/zephyr/demo_runtime/qemu-hack/qemu-system-arm",
"apps/microtvm/zephyr/demo_runtime/qemu-hack/qemu-system-riscv32",
"apps/microtvm/zephyr/demo_runtime/qemu-hack/qemu-system-riscv64",
+ # STM32 demo
Review comment:
are these linker scripts still needed?
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_create ( \n")
+ out_h.write(f" const ai_ptr weights, \n")
+ out_h.write(f" const ai_ptr activations \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_destroy (); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_create \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_create( \n")
+ out_c.write(f" const ai_ptr weights, \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" ai_status status = AI_STATUS_OK;\n")
+ out_c.write(f" status = {name}_configure_weights (weights); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" status = {name}_configure_activations (activations); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_destroy \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_destroy () \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+
+ # ==================================================================
+ # emit_code
+ # ==================================================================
+ def emit_code(self, dest_dir, model_name):
+ """ Emit the C code implementing the model. """
+
+ #
+ # Build the directory structure
+ #
+ if os.path.exists(dest_dir):
+ print(f'Removing existing "{dest_dir}" directory')
+ try:
+ shutil.rmtree(dest_dir)
+ except OSError as err:
+ raise ValueError(f"emit_code.Error: {dest_dir} : {err.strerror}")
+
+ # Make a new one
+ os.mkdir(dest_dir)
+
+ #
+ # Fix the model name
+ #
+ model_name = re.sub("[^0-9a-zA-Z_]+", "_", model_name)
+ model_name = model_name.lower()
+
+ #
+ # Write the C code: we can parse the string
+ #
+ if isinstance(self.lib_, list):
+ # List of strings from Model Library Format C files
+ for idx, src in enumerate(self.lib_):
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib{idx}.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+ # List of C files from Module Format
Review comment:
remove commented code
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_create ( \n")
+ out_h.write(f" const ai_ptr weights, \n")
+ out_h.write(f" const ai_ptr activations \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_destroy (); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_create \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_create( \n")
+ out_c.write(f" const ai_ptr weights, \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" ai_status status = AI_STATUS_OK;\n")
+ out_c.write(f" status = {name}_configure_weights (weights); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" status = {name}_configure_activations (activations); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_destroy \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_destroy () \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+
+ # ==================================================================
+ # emit_code
+ # ==================================================================
+ def emit_code(self, dest_dir, model_name):
+ """ Emit the C code implementing the model. """
+
+ #
+ # Build the directory structure
+ #
+ if os.path.exists(dest_dir):
+ print(f'Removing existing "{dest_dir}" directory')
+ try:
+ shutil.rmtree(dest_dir)
+ except OSError as err:
+ raise ValueError(f"emit_code.Error: {dest_dir} : {err.strerror}")
+
+ # Make a new one
+ os.mkdir(dest_dir)
Review comment:
you may prefer os.mkdirs() if you expect the destination dir to not exist
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
Review comment:
remove
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
Review comment:
it may be cleaner to just store `self._graph = ` here and then access these via index e.g. s/self.nodes_/self._graph["nodes"]/g`
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_create ( \n")
+ out_h.write(f" const ai_ptr weights, \n")
+ out_h.write(f" const ai_ptr activations \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_destroy (); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_create \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_create( \n")
+ out_c.write(f" const ai_ptr weights, \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" ai_status status = AI_STATUS_OK;\n")
+ out_c.write(f" status = {name}_configure_weights (weights); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" status = {name}_configure_activations (activations); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_destroy \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_destroy () \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+
+ # ==================================================================
+ # emit_code
+ # ==================================================================
+ def emit_code(self, dest_dir, model_name):
+ """ Emit the C code implementing the model. """
+
+ #
+ # Build the directory structure
+ #
+ if os.path.exists(dest_dir):
+ print(f'Removing existing "{dest_dir}" directory')
+ try:
+ shutil.rmtree(dest_dir)
+ except OSError as err:
+ raise ValueError(f"emit_code.Error: {dest_dir} : {err.strerror}")
+
+ # Make a new one
+ os.mkdir(dest_dir)
+
+ #
+ # Fix the model name
+ #
+ model_name = re.sub("[^0-9a-zA-Z_]+", "_", model_name)
+ model_name = model_name.lower()
+
+ #
+ # Write the C code: we can parse the string
+ #
+ if isinstance(self.lib_, list):
+ # List of strings from Model Library Format C files
+ for idx, src in enumerate(self.lib_):
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib{idx}.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+ # List of C files from Module Format
+ # for idx, filename in enumerate(self.lib_):
+ # with open(filename, 'r') as fin:
+ # src = fin.read()
+ # code = _preprocess_code(src)
+ # filename = os.path.join(dest_dir, f'{model_name}_lib{idx}.c')
+ # with open (filename, "w") as fout:
+ # fout.write(code)
+ else:
+ # a TVM RuntimeGraphFactory
+ # src = src + self.lib_.get_source(fmt="c")
+ # for m in self.lib_.imported_modules:
+ # src = src + m.get_source(fmt="c")
+ src = self.lib_.get_source(fmt="c")
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+
+ #
+ # Save params as bynary data
Review comment:
nit: binary
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
Review comment:
i don't see tests for scenarios when these parameters are not the defaults. could you either add tests or remove the code that runs when they're False?
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
Review comment:
while the leading `__` is technically not against python style, python makes heavy use of "dunder" functions, e.g.:
```
def __iter__(self):
"""The Python default way to iterate over an object is to call this function, the invoke __next__() on the return value."""
```
starting methods with a double-underscore like this makes the reader think they are going to be Python dunder methods. suggest changing to `_emit_create_destroy`
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
Review comment:
just unpack in the for loop:
```suggestion
for src_nid, src_idx, _ in node["inputs"]:
```
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
Review comment:
using enumerate keeps track of nid more cleanly:
```suggestion
for nid, node in enumerate(self.nodes_):
```
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
Review comment:
suggest moving these closer to where they are used, since they are just used in the one function iiuc. otherwise, can you add comments to explain their purpose?
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -0,0 +1,421 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import struct
+
+#
+# Disable GPU usage information:
+#
+#
+# 0 = all messages are logged (default behavior)
+# 1 = INFO messages are not printed
+# 2 = INFO and WARNING messages are not printed
+# 3 = INFO, WARNING, and ERROR messages are not printed
+os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
+
+import numpy as np
+
+import tensorflow as tf
+
+import tvm
+import tvm.relay as relay
+from tvm.contrib import stm32
+from tvm import testing
+
+TEST_IMAGES = [
+ "00.raw",
+ "01.raw",
+ "02.raw",
+ "03.raw",
+ "04.raw",
+ "05.raw",
+ "06.raw",
+ "07.raw",
+ "08.raw",
+ "09.raw",
+]
+
+BUILD_DIR = "build"
+
+# ==================================================================
+# dump_image
+# ==================================================================
+def dump_image(filename, image):
+ # Flatten image
+ image_data = image.flatten()
+ outputRaw = []
+ # Raw binary format
+ for i in range(0, len(image_data)):
+ outputRaw.append(struct.pack("<B", int(image_data[i]) & 0xFF))
+
+ # Dump image in raw binary format
+ f = open(filename, "wb")
+ for i in range(0, len(outputRaw)):
+ f.write(outputRaw[i])
+ f.close()
+
+
+# ==================================================================
+# scale_input_data
+# ==================================================================
+def scale_input_data(input_details, data):
+ if input_details["dtype"] == np.uint8 or input_details["dtype"] == np.int8:
+ input_scale, input_zero_point = input_details["quantization"]
+ print(
+ "== TFLite input quantization: scale={}, zero={}".format(input_scale, input_zero_point)
+ )
+ data = data / input_scale + input_zero_point
+ data = data.astype(input_details["dtype"])
+ return data
+
+
+# ==================================================================
+# scale_output_data
+# ==================================================================
+def scale_output_data(output_details, data):
+ if output_details["dtype"] == np.uint8 or output_details["dtype"] == np.int8:
+ output_scale, output_zero_point = output_details["quantization"]
+ print(
+ "== TFLite output quantization: scale={}, zero={}".format(
+ output_scale, output_zero_point
+ )
+ )
+ data = data.astype(np.float32)
+ data = (data - output_zero_point) * output_scale
+ return data
+
+
+# ========================================================
+# get_tflite_model
+# ========================================================
+def get_tflite_model(model_path):
+
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()
+ output_details = interpreter.get_output_details()
+
+ #
+ # Figure out shapes and
+ #
+
+ shape_dict = {}
+ dtype_dict = {}
+
+ for input in input_details:
+ input_name = input["name"]
+ input_shape = input["shape"].tolist()
+ input_dtype = str(np.dtype(input["dtype"]))
+ shape_dict[input_name] = input_shape
+ dtype_dict[input_name] = input_dtype
+
+ #
+ # Save the model
+ #
+
+ #
+ # Load the TFLite Model for TVM:
+ #
+ # https://docs.tvm.ai/tutorials/frontend/from_tflite.html
+ # https://jackwish.net/tflite/docs/
+
+ model_buf = open(model_path, "rb").read()
+
+ #
+ # Get TFLite model from buffer
+ #
+ try:
+ import tflite
+
+ model = tflite.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model)
+ except AttributeError:
+ import tflite.Model
+
+ model = tflite.Model.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model.Model)
+
+ print("TVM: Importing a TFLite model ...")
+
+ return model, shape_dict, dtype_dict
+
+
+# ========================================================
+# extract_tflite_quantization
+# ========================================================
+
+
+def _make_qnn_params(quantization):
+ qnn_params = {}
+ qnn_params["min"] = quantization.MinAsNumpy()
+ qnn_params["max"] = quantization.MaxAsNumpy()
+ qnn_params["scale"] = quantization.ScaleAsNumpy()
+ qnn_params["zero_point"] = quantization.ZeroPointAsNumpy()
+ qnn_params["dim"] = quantization.QuantizedDimension()
+ # print(" Quantization: ({}, {}), s={}, z={}, dim={}".format(min, max, scale, zero_point, dim))
+ return qnn_params
+
+
+def extract_tflite_quantization(model):
+
+ assert model.SubgraphsLength() == 1, "only support one subgraph (main subgraph)"
+
+ subgraph = model.Subgraphs(0)
+
+ quantization_info = {}
+
+ # model inputs / outputs
+ model_inputs = subgraph.InputsAsNumpy()
+ model_outputs = subgraph.OutputsAsNumpy()
+
+ for node_id in model_inputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ # print("== Input[{}]: {} shape={} type={}".format(node_id, tensor_name, tensor.ShapeAsNumpy(), tensor_type))
+ dl_tensor_name = stm32.get_input_tensor_name(tensor_name)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ for node_id in model_outputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ # print("== Output[{}]: {} shape={} type={}".format(node_id, tensor_name, tensor.ShapeAsNumpy(), tensor_type))
+ #
+ # TODO: TVM does not preserve the output tensor names.
+ # Eventually, we should be able to form a valid name.
+ #
+ dl_tensor_name = stm32.get_output_tensor_name(tensor_name, 0)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ return quantization_info
+
+
+# ========================================================
+# run_tflite_model
+# ========================================================
+def run_tflite_model(model_path, image_path):
+
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()[0]
+ output_details = interpreter.get_output_details()[0]
+
+ #
+ # Run test images
+ #
+ tf_results = np.empty(shape=[len(TEST_IMAGES), 10], dtype=np.float)
+ for i, filename in enumerate(TEST_IMAGES):
+ image_data = np.fromfile(os.path.join(image_path, filename), dtype="uint8")
+ #
+ # Run the TFLite model: channels last
+ #
+ image_data = image_data.reshape([1, 28, 28, 1])
+ #
+ # Normalize the input data
+ #
+ image_data = image_data / 255.0
+
+ image_data = scale_input_data(input_details, image_data)
+ interpreter.set_tensor(input_details["index"], image_data)
+ interpreter.invoke()
+ tf_results[i] = interpreter.get_tensor(output_details["index"])
+ tf_results[i] = scale_output_data(output_details, tf_results[i])
+
+ print(f"== [{i}] TFLite Output:")
+ print(tf_results[i])
+
+ return tf_results
+
+
+# ========================================================
+# run_tvm_model
+# ========================================================
+def run_tvm_model(build_dir, model_name, target_dir, image_path):
+
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+
+ tvm_results_name = os.path.join(build_dir, "tvm_results.txt")
+
+ #
+ # Build the model
+ #
+ tvm_dir = os.path.join(curr_path, "..", "..", "..")
+ test_dir = os.path.join(tvm_dir, "tests", "crt", "contrib", "stm32")
+
+ command = f"make -f {test_dir}/Makefile TVM_PATH={tvm_dir} MODEL_PATH={target_dir} BUILD_PATH={build_dir} IMAGE_PATH={image_path}"
+ print(f"{command}")
+ os.system(command)
+ #
+ # Run
+ #
+ command = f"{target_dir}/{model_name}.exe"
+ print(f"{command}")
+ os.system(command)
+
+ tvm_results = np.loadtxt(tvm_results_name)
+ print(f"== TVM Output:\n {tvm_results}")
+
+ #
+ # Clean temporary image files
+ #
+ if os.path.exists(tvm_results_name):
+ os.remove(tvm_results_name)
+
+ return tvm_results
+
+
+# ========================================================
+# check_network
+# ========================================================
+def check_network(build_dir, target_name, model_path, image_path):
+
+ model_name = "network"
+
+ target_dir = os.path.join(build_dir, target_name + "_gen")
+
+ model, shape_dict, dtype_dict = get_tflite_model(model_path)
+ mod, params = relay.frontend.from_tflite(model, shape_dict, dtype_dict)
+
+ #
+ # Build a TVM C module for the ARM CPU (without compiling the kernels
+ # library to the object code form):
+ #
+ target = "c -device=arm_cpu"
+ opt_level = 3
+
+ with tvm.transform.PassContext(opt_level=opt_level, config={"tir.disable_vectorize": True}):
+ rt_module = relay.build(mod, target=target, params=params)
+
+ #
+ # Export model library format
+ #
+ mlf_tar_path = os.path.join(build_dir, target_name + "_lib.tar")
+ import tvm.micro as micro
+
+ micro.export_model_library_format(rt_module, mlf_tar_path)
+
+ emitter = stm32.CodeEmitter()
+ quantization = extract_tflite_quantization(model)
+ emitter.parse_library_format(mlf_tar_path, quantization)
+ emitter.emit_code(target_dir, model_name)
+
+ #
+ # Results
+ #
+ tf_results = run_tflite_model(model_path, image_path)
+ tvm_results = run_tvm_model(build_dir, model_name, target_dir, image_path)
+
+ check_result(tf_results, tvm_results)
+
+
+# ========================================================
+# check_result
+# ========================================================
+def check_result(tflite_results, tvm_results):
+ """Helper function to verify results"""
+
+ #
+ # MNIST quantized uint8 results in one single difference of
+ # ~ 0.004 so just escape this
+ #
+ ATOL = 1e-3
+ RTOL = 0.5
+
+ tvm.testing.assert_allclose(tflite_results, tvm_results, rtol=RTOL, atol=ATOL)
+
+
+# ========================================================
+# test_mnist_quant_fp
+# ========================================================
+def test_mnist_quant_fp():
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+ model_path = os.path.join(curr_path, "models/mnist.quant.2_mod.tflite")
+ image_path = os.path.join(curr_path, "images")
+ build_dir = os.path.join(curr_path, BUILD_DIR)
+ if not os.path.exists(build_dir):
+ os.makedirs(build_dir)
+ check_network(build_dir, "mnist_quant_fp", model_path, image_path)
+
+
+# ========================================================
+# test_mnist_quant_int
+# ========================================================
+def test_mnist_quant_int():
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+ model_path = os.path.join(curr_path, "models/mnist_q_with_int8_io.tflite")
+ image_path = os.path.join(curr_path, "images")
+ build_dir = os.path.join(curr_path, BUILD_DIR)
+ if not os.path.exists(build_dir):
+ os.makedirs(build_dir)
+ check_network(build_dir, "mnist_quant_int", model_path, image_path)
+
+
+# ========================================================
+# test_mnist_quant_uint
+# ========================================================
+def test_mnist_quant_uint():
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+ model_path = os.path.join(curr_path, "models/mnist_q_with_uint8_io.tflite")
+ image_path = os.path.join(curr_path, "images")
+ build_dir = os.path.join(curr_path, BUILD_DIR)
+ if not os.path.exists(build_dir):
+ os.makedirs(build_dir)
+ check_network(build_dir, "mnist_quant_uint", model_path, image_path)
+
+
+# ========================================================
+# test_mnist_fp
+# ========================================================
+def test_mnist_fp():
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+ model_path = os.path.join(curr_path, "models/mnist.tflite")
+ image_path = os.path.join(curr_path, "images")
+ build_dir = os.path.join(curr_path, BUILD_DIR)
+ if not os.path.exists(build_dir):
+ os.makedirs(build_dir)
+ check_network(build_dir, "mnist_fp", model_path, image_path)
+
+
+if __name__ == "__main__":
+ # test_mnist_fp()
Review comment:
remove
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
Review comment:
remove commented code
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
Review comment:
```suggestion
assert dl_tensor_name in self.input_data_, f"Tensor {dl_tensor_name} not registered ?"
```
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
Review comment:
specify meaning of XXX
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_create ( \n")
+ out_h.write(f" const ai_ptr weights, \n")
+ out_h.write(f" const ai_ptr activations \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_destroy (); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_create \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_create( \n")
+ out_c.write(f" const ai_ptr weights, \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" ai_status status = AI_STATUS_OK;\n")
+ out_c.write(f" status = {name}_configure_weights (weights); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" status = {name}_configure_activations (activations); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_destroy \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_destroy () \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+
+ # ==================================================================
+ # emit_code
+ # ==================================================================
+ def emit_code(self, dest_dir, model_name):
+ """ Emit the C code implementing the model. """
+
+ #
+ # Build the directory structure
+ #
+ if os.path.exists(dest_dir):
+ print(f'Removing existing "{dest_dir}" directory')
+ try:
+ shutil.rmtree(dest_dir)
+ except OSError as err:
+ raise ValueError(f"emit_code.Error: {dest_dir} : {err.strerror}")
+
+ # Make a new one
+ os.mkdir(dest_dir)
+
+ #
+ # Fix the model name
+ #
+ model_name = re.sub("[^0-9a-zA-Z_]+", "_", model_name)
+ model_name = model_name.lower()
+
+ #
+ # Write the C code: we can parse the string
+ #
+ if isinstance(self.lib_, list):
+ # List of strings from Model Library Format C files
+ for idx, src in enumerate(self.lib_):
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib{idx}.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+ # List of C files from Module Format
+ # for idx, filename in enumerate(self.lib_):
+ # with open(filename, 'r') as fin:
+ # src = fin.read()
+ # code = _preprocess_code(src)
+ # filename = os.path.join(dest_dir, f'{model_name}_lib{idx}.c')
+ # with open (filename, "w") as fout:
+ # fout.write(code)
+ else:
+ # a TVM RuntimeGraphFactory
+ # src = src + self.lib_.get_source(fmt="c")
+ # for m in self.lib_.imported_modules:
+ # src = src + m.get_source(fmt="c")
+ src = self.lib_.get_source(fmt="c")
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+
+ #
+ # Save params as bynary data
+ #
+ saved_params = tvm.runtime.save_param_dict(self.params_)
+ params_name = os.path.join(dest_dir, model_name + ".params")
+ with open(params_name, "wb") as f:
+ f.write(saved_params)
+
+ #
+ # Write the .json
+ #
+ graph_name = os.path.join(dest_dir, model_name + ".json")
+ with open(graph_name, "w") as f:
+ json.dump(self.graph_, f)
+
+ #
+ # emit X_data[c,h]
+ #
+
+ data_h_name = os.path.join(dest_dir, model_name + "_data.h")
Review comment:
if these are just passed to open(), suggest just placing them there for readability. also suggest using `with` blocks to ensure files are closed when error conditions occur:
you may also find pathlib makes `os.path.join` easier to read:
```
dest_dir = pathlib.Path(dest_dir)
with open(dest_dir / f"{model_name}_data.h") as data_h:
with open(dest_dir / f"{model_name}_data.c") as data_c:
self.__emit_params_data(model_name, data_h, data_c)
```
this style also encourages keeping as few files open as are necessary to avoid excessive indent
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -0,0 +1,421 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import sys
+import struct
+
+#
+# Disable GPU usage information:
+#
+#
+# 0 = all messages are logged (default behavior)
+# 1 = INFO messages are not printed
+# 2 = INFO and WARNING messages are not printed
+# 3 = INFO, WARNING, and ERROR messages are not printed
+os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
+
+import numpy as np
+
+import tensorflow as tf
+
+import tvm
+import tvm.relay as relay
+from tvm.contrib import stm32
+from tvm import testing
+
+TEST_IMAGES = [
+ "00.raw",
+ "01.raw",
+ "02.raw",
+ "03.raw",
+ "04.raw",
+ "05.raw",
+ "06.raw",
+ "07.raw",
+ "08.raw",
+ "09.raw",
+]
+
+BUILD_DIR = "build"
+
+# ==================================================================
+# dump_image
+# ==================================================================
+def dump_image(filename, image):
+ # Flatten image
+ image_data = image.flatten()
+ outputRaw = []
+ # Raw binary format
+ for i in range(0, len(image_data)):
+ outputRaw.append(struct.pack("<B", int(image_data[i]) & 0xFF))
+
+ # Dump image in raw binary format
+ f = open(filename, "wb")
+ for i in range(0, len(outputRaw)):
+ f.write(outputRaw[i])
+ f.close()
+
+
+# ==================================================================
+# scale_input_data
+# ==================================================================
+def scale_input_data(input_details, data):
+ if input_details["dtype"] == np.uint8 or input_details["dtype"] == np.int8:
+ input_scale, input_zero_point = input_details["quantization"]
+ print(
+ "== TFLite input quantization: scale={}, zero={}".format(input_scale, input_zero_point)
+ )
+ data = data / input_scale + input_zero_point
+ data = data.astype(input_details["dtype"])
+ return data
+
+
+# ==================================================================
+# scale_output_data
+# ==================================================================
+def scale_output_data(output_details, data):
+ if output_details["dtype"] == np.uint8 or output_details["dtype"] == np.int8:
+ output_scale, output_zero_point = output_details["quantization"]
+ print(
+ "== TFLite output quantization: scale={}, zero={}".format(
+ output_scale, output_zero_point
+ )
+ )
+ data = data.astype(np.float32)
+ data = (data - output_zero_point) * output_scale
+ return data
+
+
+# ========================================================
+# get_tflite_model
+# ========================================================
+def get_tflite_model(model_path):
+
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()
+ output_details = interpreter.get_output_details()
+
+ #
+ # Figure out shapes and
+ #
+
+ shape_dict = {}
+ dtype_dict = {}
+
+ for input in input_details:
+ input_name = input["name"]
+ input_shape = input["shape"].tolist()
+ input_dtype = str(np.dtype(input["dtype"]))
+ shape_dict[input_name] = input_shape
+ dtype_dict[input_name] = input_dtype
+
+ #
+ # Save the model
+ #
+
+ #
+ # Load the TFLite Model for TVM:
+ #
+ # https://docs.tvm.ai/tutorials/frontend/from_tflite.html
+ # https://jackwish.net/tflite/docs/
+
+ model_buf = open(model_path, "rb").read()
+
+ #
+ # Get TFLite model from buffer
+ #
+ try:
+ import tflite
+
+ model = tflite.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model)
+ except AttributeError:
+ import tflite.Model
+
+ model = tflite.Model.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model.Model)
+
+ print("TVM: Importing a TFLite model ...")
+
+ return model, shape_dict, dtype_dict
+
+
+# ========================================================
+# extract_tflite_quantization
+# ========================================================
+
+
+def _make_qnn_params(quantization):
+ qnn_params = {}
+ qnn_params["min"] = quantization.MinAsNumpy()
+ qnn_params["max"] = quantization.MaxAsNumpy()
+ qnn_params["scale"] = quantization.ScaleAsNumpy()
+ qnn_params["zero_point"] = quantization.ZeroPointAsNumpy()
+ qnn_params["dim"] = quantization.QuantizedDimension()
+ # print(" Quantization: ({}, {}), s={}, z={}, dim={}".format(min, max, scale, zero_point, dim))
+ return qnn_params
+
+
+def extract_tflite_quantization(model):
+
+ assert model.SubgraphsLength() == 1, "only support one subgraph (main subgraph)"
+
+ subgraph = model.Subgraphs(0)
+
+ quantization_info = {}
+
+ # model inputs / outputs
+ model_inputs = subgraph.InputsAsNumpy()
+ model_outputs = subgraph.OutputsAsNumpy()
+
+ for node_id in model_inputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ # print("== Input[{}]: {} shape={} type={}".format(node_id, tensor_name, tensor.ShapeAsNumpy(), tensor_type))
+ dl_tensor_name = stm32.get_input_tensor_name(tensor_name)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ for node_id in model_outputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ # print("== Output[{}]: {} shape={} type={}".format(node_id, tensor_name, tensor.ShapeAsNumpy(), tensor_type))
+ #
+ # TODO: TVM does not preserve the output tensor names.
+ # Eventually, we should be able to form a valid name.
+ #
+ dl_tensor_name = stm32.get_output_tensor_name(tensor_name, 0)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ return quantization_info
+
+
+# ========================================================
+# run_tflite_model
+# ========================================================
+def run_tflite_model(model_path, image_path):
+
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()[0]
+ output_details = interpreter.get_output_details()[0]
+
+ #
+ # Run test images
+ #
+ tf_results = np.empty(shape=[len(TEST_IMAGES), 10], dtype=np.float)
+ for i, filename in enumerate(TEST_IMAGES):
+ image_data = np.fromfile(os.path.join(image_path, filename), dtype="uint8")
+ #
+ # Run the TFLite model: channels last
+ #
+ image_data = image_data.reshape([1, 28, 28, 1])
+ #
+ # Normalize the input data
+ #
+ image_data = image_data / 255.0
+
+ image_data = scale_input_data(input_details, image_data)
+ interpreter.set_tensor(input_details["index"], image_data)
+ interpreter.invoke()
+ tf_results[i] = interpreter.get_tensor(output_details["index"])
+ tf_results[i] = scale_output_data(output_details, tf_results[i])
+
+ print(f"== [{i}] TFLite Output:")
+ print(tf_results[i])
+
+ return tf_results
+
+
+# ========================================================
+# run_tvm_model
+# ========================================================
+def run_tvm_model(build_dir, model_name, target_dir, image_path):
+
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+
+ tvm_results_name = os.path.join(build_dir, "tvm_results.txt")
+
+ #
+ # Build the model
+ #
+ tvm_dir = os.path.join(curr_path, "..", "..", "..")
+ test_dir = os.path.join(tvm_dir, "tests", "crt", "contrib", "stm32")
+
+ command = f"make -f {test_dir}/Makefile TVM_PATH={tvm_dir} MODEL_PATH={target_dir} BUILD_PATH={build_dir} IMAGE_PATH={image_path}"
+ print(f"{command}")
+ os.system(command)
+ #
+ # Run
+ #
+ command = f"{target_dir}/{model_name}.exe"
+ print(f"{command}")
+ os.system(command)
+
+ tvm_results = np.loadtxt(tvm_results_name)
+ print(f"== TVM Output:\n {tvm_results}")
+
+ #
+ # Clean temporary image files
+ #
+ if os.path.exists(tvm_results_name):
+ os.remove(tvm_results_name)
+
+ return tvm_results
+
+
+# ========================================================
+# check_network
+# ========================================================
+def check_network(build_dir, target_name, model_path, image_path):
+
+ model_name = "network"
+
+ target_dir = os.path.join(build_dir, target_name + "_gen")
+
+ model, shape_dict, dtype_dict = get_tflite_model(model_path)
+ mod, params = relay.frontend.from_tflite(model, shape_dict, dtype_dict)
+
+ #
+ # Build a TVM C module for the ARM CPU (without compiling the kernels
+ # library to the object code form):
+ #
+ target = "c -device=arm_cpu"
+ opt_level = 3
+
+ with tvm.transform.PassContext(opt_level=opt_level, config={"tir.disable_vectorize": True}):
+ rt_module = relay.build(mod, target=target, params=params)
+
+ #
+ # Export model library format
+ #
+ mlf_tar_path = os.path.join(build_dir, target_name + "_lib.tar")
+ import tvm.micro as micro
+
+ micro.export_model_library_format(rt_module, mlf_tar_path)
+
+ emitter = stm32.CodeEmitter()
+ quantization = extract_tflite_quantization(model)
+ emitter.parse_library_format(mlf_tar_path, quantization)
+ emitter.emit_code(target_dir, model_name)
+
+ #
+ # Results
+ #
+ tf_results = run_tflite_model(model_path, image_path)
+ tvm_results = run_tvm_model(build_dir, model_name, target_dir, image_path)
+
+ check_result(tf_results, tvm_results)
+
+
+# ========================================================
+# check_result
+# ========================================================
+def check_result(tflite_results, tvm_results):
+ """Helper function to verify results"""
+
+ #
+ # MNIST quantized uint8 results in one single difference of
+ # ~ 0.004 so just escape this
+ #
+ ATOL = 1e-3
+ RTOL = 0.5
+
+ tvm.testing.assert_allclose(tflite_results, tvm_results, rtol=RTOL, atol=ATOL)
+
+
+# ========================================================
+# test_mnist_quant_fp
+# ========================================================
+def test_mnist_quant_fp():
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+ model_path = os.path.join(curr_path, "models/mnist.quant.2_mod.tflite")
Review comment:
we need to avoid repo bloat by not adding large test data (this model is MBs). can you download this from somewhere? see `download_testdata` in `python/tvm/contrib/download.py`.
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
Review comment:
remove
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
Review comment:
```suggestion
input_idx = self.input_data_.find(dl_tensor_name)
assert input_idx >= 0, "..."
```
##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -190,6 +190,7 @@ TVM_DLL void TVMAPISetLastError(const char* msg);
* \return error info
*/
TVM_DLL const char* TVMGetLastError(void);
+
Review comment:
remove
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
Review comment:
remove commented code
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
+
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ #
+ # call this function
+ #
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} ... \\r\\n"); \n')
+ out_c.write(f"#endif \n")
+
+ out_c.write(
+ f" if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{ \n"
+ )
+ out_c.write(f' TVMAPISetLastError("Invalid handle");\n')
+ out_c.write(f" return AI_STATUS_ERROR; \n")
+
+ out_c.write(f" }} \n")
+ out_c.write(f"#if (_VERBOSE_ > 0) \n")
+ out_c.write(f' printf (" {func_name} Done.\\r\\n"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"#ifdef _DUMP_{node_name_upper}_ \n")
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ if dl_tensor_name in self.activations_:
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ else:
+ assert dl_tensor_name in self.output_data_
+ output_idx = 0
+ for dl_exit_name in self.output_data_:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", outputs[{output_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+
+ # nid += 1
+
+ out_c.write(f"\n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f" fclose(DumpFile_p); \n")
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_create_destroy
+ # ==========================================================
+ def __emit_create_destroy(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_create ( \n")
+ out_h.write(f" const ai_ptr weights, \n")
+ out_h.write(f" const ai_ptr activations \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_destroy (); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_create \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_create( \n")
+ out_c.write(f" const ai_ptr weights, \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" ai_status status = AI_STATUS_OK;\n")
+ out_c.write(f" status = {name}_configure_weights (weights); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" status = {name}_configure_activations (activations); \n")
+ out_c.write(f" if (status != AI_STATUS_OK) {{\n")
+ out_c.write(f" return status;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_destroy \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_destroy () \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+
+ # ==================================================================
+ # emit_code
+ # ==================================================================
+ def emit_code(self, dest_dir, model_name):
+ """ Emit the C code implementing the model. """
+
+ #
+ # Build the directory structure
+ #
+ if os.path.exists(dest_dir):
+ print(f'Removing existing "{dest_dir}" directory')
+ try:
+ shutil.rmtree(dest_dir)
+ except OSError as err:
+ raise ValueError(f"emit_code.Error: {dest_dir} : {err.strerror}")
+
+ # Make a new one
+ os.mkdir(dest_dir)
+
+ #
+ # Fix the model name
+ #
+ model_name = re.sub("[^0-9a-zA-Z_]+", "_", model_name)
+ model_name = model_name.lower()
+
+ #
+ # Write the C code: we can parse the string
+ #
+ if isinstance(self.lib_, list):
+ # List of strings from Model Library Format C files
+ for idx, src in enumerate(self.lib_):
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib{idx}.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+ # List of C files from Module Format
+ # for idx, filename in enumerate(self.lib_):
+ # with open(filename, 'r') as fin:
+ # src = fin.read()
+ # code = _preprocess_code(src)
+ # filename = os.path.join(dest_dir, f'{model_name}_lib{idx}.c')
+ # with open (filename, "w") as fout:
+ # fout.write(code)
+ else:
+ # a TVM RuntimeGraphFactory
+ # src = src + self.lib_.get_source(fmt="c")
+ # for m in self.lib_.imported_modules:
+ # src = src + m.get_source(fmt="c")
+ src = self.lib_.get_source(fmt="c")
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+
+ #
+ # Save params as bynary data
+ #
+ saved_params = tvm.runtime.save_param_dict(self.params_)
+ params_name = os.path.join(dest_dir, model_name + ".params")
+ with open(params_name, "wb") as f:
+ f.write(saved_params)
+
+ #
+ # Write the .json
+ #
+ graph_name = os.path.join(dest_dir, model_name + ".json")
Review comment:
possible to just `shutil.copy2()` from the MLF archive, here and above?
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==================================================================
+ # _json_load_attrs
+ # ==================================================================
+ # def __json_load_attrs (self, attr_dict):
+
+ # param = None
+
+ # for key in attr_dict:
+
+ # if key == 'func_name':
+ # param->func_name = value;
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_inputs':
+ # param->num_inputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'num_outputs':
+ # param->num_outputs = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+ # elif key == 'flatten_data':
+ # param->flatten_data = strtoul(value.c_str(), nullptr, 10);
+ # print ("json_load_attrs: TODO.")
+
+ # return param
+
+ # ==========================================================
+ # _json_parse_nodes
+ # ==========================================================
+ # def __json_parse_nodes(self, nodes):
+ # """ Parse the nodes part of the JSON graph. """
+
+ # for node_dict in nodes:
+
+ # print (" node: {}".format(node_dict))
+
+ # op = None
+ # name = None
+ # inputs = None
+ # attrs = None
+ # deps = None
+
+ # for key in node_dict:
+
+ # if key == 'op':
+ # op = node_dict['op']
+ # print (" op: {}".format(node_dict['op']))
+ # elif key == 'name':
+ # name = node_dict['name']
+ # print (" name: {}".format(node_dict['name']))
+ # elif key == 'inputs':
+ # inputs = node_dict['inputs']
+ # print (" inputs: {}".format(node_dict['inputs']))
+ # elif key == 'attr':
+ # attrs = node_dict['attr']
+ # elif key == 'attrs':
+ # attrs = node_dict['attrs']
+ # elif key == 'control_deps':
+ # deps = node_dict['control_deps']
+ # else:
+ # print("### Error: JSON do not support key {}".format(key))
+ # assert False
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ # out_version = out[2]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ # shape = str(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ node_name = node["name"]
+
+ # print ("== [{}] op: {}".format(nid, node['name']))
+
+ if node["op"] != "null":
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_attrs = node["attrs"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+ # version = output[2]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ # print (" -- nodes: {}".format(graph_dict['nodes']))
+ # self.__json_parse_nodes(self.nodes_)
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ # print (" -- inputs: {}".format(arg_nodes_))
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ # print (" -- node_row_ptr: {}".format(graph_dict['node_row_ptr']))
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ # print (" -- outputs: {}".format(outputs_))
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ # print (" -- attrs: {}".format(attrs_))
+ elif key == "metadata":
+ # print (" -- meta: {}".format(graph_dict['metadata']))
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+ # for filename in os.listdir(src_dir):
+ # if filename.endswith(".c"):
+ # src_files.append(os.path.join(src_dir, filename))
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict # module.get_params()
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # __emit_params_data
+ # ==========================================================
+ def __emit_params_data(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(f"#ifndef __{name_upper}_DATA_H_ \n")
+ out_h.write(f"#define __{name_upper}_DATA_H_ \n")
+ out_h.write(f"\n")
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"const ai_ptr ai_{name}_data_weights_get (void); \n")
+ out_h.write(f"\n")
+
+ out_h.write(f"#endif /* __{name_upper}_DATA_H_ */ \n")
+ out_h.write(f"\n")
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ out_c.write(f"const ai_ptr ai_{name}_data_weights_get (void) \n")
+ out_c.write(f"{{\n")
+ out_c.write(
+ f' AI_ALIGNED({self.data_alignment}) static const __attribute__ ((section(".nn_weights"))) uint8_t s_{name}_weights[] = {{ \n'
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(f" }}; \n")
+ out_c.write(f" return (const ai_ptr)s_{name}_weights; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_open
+ # ==========================================================
+ def __emit_open(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#ifndef __AI_{name_upper}_H__ \n")
+ out_h.write(f"#define __AI_{name_upper}_H__ \n")
+ out_h.write(f"\n")
+
+ out_h.write(f'#include "ai_runtime_api.h" \n')
+ out_h.write(f"\n")
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+ out_h.write(f"#define _{name_upper}_INPUTS_COUNT_ ({input_size})\n")
+ out_h.write(f"#define _{name_upper}_OUTPUTS_COUNT_ ({output_size})\n")
+ out_h.write(f"#define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})\n")
+ out_h.write(f"\n")
+
+ #
+ # XXX.c
+ #
+ out_c.write(f"#include <stdio.h> \n")
+ out_c.write(f"\n")
+
+ out_c.write(f'#include "dlpack/dlpack.h" \n')
+ out_c.write(f'#include "tvm/runtime/c_runtime_api.h" \n')
+ # out_c.write(f'#include "tvm/runtime/c_backend_api.h" \n')
+ out_c.write(f'#include "{name}.h" \n')
+ out_c.write(f'#include "{name}_data.h" \n')
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_close
+ # ==========================================================
+ def __emit_close(self, name, out_h, out_c):
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+ # print("date and time =", dt_string)
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ out_c.write(
+ f'AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{\n'
+ )
+ out_c.write(f' .name = "{name}",\n')
+ out_c.write(f' .datetime = "{dt_string}",\n')
+ out_c.write(f' .revision = "{AI_TOOLS_REVISION}",\n')
+ out_c.write(
+ f' .tool_version = "{AI_TOOLS_VERSION_MAJOR}.{AI_TOOLS_VERSION_MINOR}.{AI_TOOLS_VERSION_MICRO}.0",\n'
+ )
+ out_c.write(
+ f' .api_version = "{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0",\n'
+ )
+ # out_c.write (" .n_macc = {},\n".format(0))
+ out_c.write(f" .n_nodes = {self.nodes_size_},\n")
+ out_c.write(f" .n_inputs = {num_inputs},\n")
+ out_c.write(f" .n_outputs = {num_outputs},\n")
+ out_c.write(f" .activations_size = {self.activations_size_},\n")
+ out_c.write(f" .params_size = {self.weights_size_},\n")
+ if self.activations_static == 1:
+ out_c.write(f" .activations = {name}_activations, \n")
+ else:
+ out_c.write(f" .activations = NULL, \n")
+ out_c.write(f" .inputs = _InputsList,\n")
+ out_c.write(f" .outputs = _OutputsList,\n")
+ out_c.write(f" .ai_get_params = &ai_{name}_data_weights_get,\n")
+ out_c.write(f" .ai_create = &ai_{name}_create,\n")
+ out_c.write(f" .ai_destroy = &ai_{name}_destroy,\n")
+ out_c.write(f" .ai_run = &ai_{name}_run\n")
+ out_c.write(f"}};\n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_shape
+ # ==========================================================
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ # ==========================================================
+ # __emit_tensor_quant
+ # ==========================================================
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ # minval = quantization['min']
+ # maxval = quantization['max']
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+ # dim = quantization['dim']
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ # print ("== Setting quantization for {}: {}".format(dl_tensor_name, quantization))
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC ai_quantization_info {quant_name} = {{\n")
+ out_c.write(f" .scale = {quant_name}_scale,\n")
+ out_c.write(f" .zero_point = {quant_name}_zero_point,\n")
+ out_c.write(f" .dim = -1\n")
+ out_c.write(f"}}; \n")
+
+ return quant_name
+
+ return None
+
+ # ==========================================================
+ # __emit_tensor_init
+ # ==========================================================
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ # storage_id = tensor['storage_id']
+ byte_offset = tensor["byte_offset"]
+ # size = tensor['size']
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ out_c.write(
+ f"AI_ALIGNED({self.data_alignment}) AI_STATIC ai_tensor {dl_tensor_name} = {{ \n"
+ )
+ out_c.write(f" .dltensor = {{ \n")
+ out_c.write(f" .data = (ai_ptr)(NULL), \n")
+ #
+ # TODO: use the 'storage_id'
+ #
+ # out_c.write (" .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1]))
+ out_c.write(f" .device = {{ kDLCPU, 0 }}, \n")
+ out_c.write(f" .ndim = {ndim}, \n")
+ out_c.write(f" .dtype = {{ {dtype} }}, \n")
+ out_c.write(f" .shape = {dl_tensor_name}_shape, \n")
+ if strides is not None:
+ out_c.write(f" .strides = {dl_tensor_name}_strides, \n")
+ else:
+ out_c.write(f" .strides = NULL, \n")
+ out_c.write(f" .byte_offset = {byte_offset} \n")
+ out_c.write(f" }}, \n")
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ # ==========================================================
+ # __emit_activation_buffers
+ # ==========================================================
+ def __emit_activation_buffers(self, name, out_c):
+ #
+ # Inputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Inputs:\n")
+ out_c.write(f"// \n")
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ #
+ # Activations:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Activations: \n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(f"// \n")
+ out_c.write(f"// Outputs:\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ if self.activations_static == 1:
+ out_c.write(
+ f'AI_ALIGNED({self.data_alignment}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_buffers
+ # ==========================================================
+ def __emit_params_buffers(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Weights: {name}\n")
+ out_c.write(f"// \n")
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_network
+ # ==========================================================
+ def __emit_network(self, name, out_c):
+
+ out_c.write(f"// \n")
+ out_c.write(f"// Network: {name}\n")
+ out_c.write(f"// \n")
+ #
+ # Emit prototypes for the TVM library functions
+ #
+ for node in self.nodes_:
+ if node["op"] != "null":
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_tensor_activation
+ # ==========================================================
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}: storage_id:{storage_id}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset}); \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_activation_init
+ # ==========================================================
+ def __emit_activation_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_activations \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_activations ( \n")
+ out_c.write(f" const ai_ptr activations \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (activations == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null activations arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_params_init
+ # ==========================================================
+ def __emit_params_init(self, name, out_c):
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// {name}_configure_weights \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_STATIC AI_INLINE\n")
+ out_c.write(f"ai_status {name}_configure_weights ( \n")
+ out_c.write(f" const ai_ptr weights \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+ out_c.write(f" if (weights == NULL) {{\n")
+ out_c.write(
+ f' TVMAPISetLastError ("Non-null weights arena is required for this model.");\n'
+ )
+ out_c.write(f" return AI_STATUS_ERROR;\n")
+ out_c.write(f" }}\n")
+ out_c.write(f"\n")
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(f" // \n")
+ out_c.write(f" // {dl_tensor_name}\n")
+ out_c.write(f" // \n")
+ out_c.write(f" {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset}); \n")
+ out_c.write(f"\n")
+
+ out_c.write(f" return AI_STATUS_OK; \n")
+ out_c.write(f"}} \n")
+ out_c.write(f"\n")
+
+ # ==========================================================
+ # __emit_init
+ # ==========================================================
+ def __emit_init(self, name, out_c):
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ # ==========================================================
+ # __emit_run
+ # ==========================================================
+ def __emit_run(self, name, out_h, out_c):
+
+ out_h.write(f"AI_API_ENTRY \n")
+ out_h.write(f"ai_status ai_{name}_run ( \n")
+ out_h.write(f" ai_tensor *inputs[], \n")
+ out_h.write(f" ai_tensor *outputs[] \n")
+ out_h.write(f"); \n")
+ out_h.write(f"\n")
+
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"// ai_{name}_run \n")
+ out_c.write(f"// {DBAR} \n")
+ out_c.write(f"AI_API_ENTRY \n")
+ out_c.write(f"ai_status ai_{name}_run ( \n")
+ out_c.write(f" ai_tensor *inputs[], \n")
+ out_c.write(f" ai_tensor *outputs[] \n")
+ out_c.write(f") \n")
+ out_c.write(f"{{ \n")
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ # print (" [{}] op: {}".format(nid, node['name']))
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for node_in in node["inputs"]:
+ src_nid = node_in[0]
+ src_idx = node_in[1]
+ # src_version = node_in[2]
+ src_node = self.nodes_[src_nid]
+ src_name = self.nodes_[src_nid]["name"]
+
+ if src_node["op"] == "null":
+ dl_tensor_name = get_input_tensor_name(src_name)
+ else:
+ dl_tensor_name = get_output_tensor_name(src_name, src_idx)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ elif dl_tensor_name in self.input_data_:
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", inputs[{input_idx}], DumpFile_p); \n'
+ )
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ #
+ # If this output is not an activation => find the output
+ #
+ if dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.output_data_
Review comment:
add a suitable error message here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r630528798
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
Hello, Andrew @areusch
My concern is imposing to implement the TVMLogf and the TVMPlatformAbort. While we implement the TVMPlatformAlloc as part of the stm32 runtime (this is a fairly common functionality across various boards), the TVMLogf and the TVMPlatformAbort are different. They are really board and even application-specific. We would like to avoid having hooks from the runtime back to the application. Returning the error codes from the crt_backend_api seems to be our commonly preferred way of handling errors here - I am very interested in this being proposed for the mTVM.
> We'd have to generate error-checking code in operator implementations, since that is where TVMBackendAllocWorkspace is called from. It's hard to generate macros such as CHECK_EQ there.
I am not sure I understand this point. If the memory allocation fails, we have 2 choices:
- The operator code catches it and SetLastError(), eventually with a detailed error code. I do not see why this is difficult or not clean - but perhaps I do not have enough vision here. In any case, this seems to have already been implemented with the TVM operators, Is this going to change ? Which RFC ?
- The crt layer catches the error and handles via further hooks into the application code. We prefer to minimize such hooks.
I am not sure why you have mentioned the CHECK_EQ macro ?
So, I am not sure how I should proceed with the test. I could do this:
1. I move the Set/GetLatError from the crt_runtime_api.c to the crt_backend_api.c.
2. We include the crt_backend_api.c with our test. The TVMLogf and the TVMPlatformAbort will be implemented 'dummy' with the stm32 runtime.
Is this gonna work for you ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-922663204
@stoa let me know when you're ready for another review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r716186330
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
Review comment:
Removed this comment - not important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-869902846
@stoa I've taken a look at your `c_backend_api`/`c_runtime_api` reorganization. I think it actually makes sense, but it's a bit more extensive than I originally imagined (I thought we were just discussing moving `TVMAPISetLastError` functionality). I think it would be best to open a separate PR for that, as we'll need to loop in some of the core committers for that, and this PR's thread is quite long. Would you be up for doing that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r626494235
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
How are you, Andrew @areusch
I have reorganized the code like so:
1. Removed the `apps/stm32`
2. Removed linker script code from the `emitter.py`
3. Removed the copy of `graph_runtime` from `src/runtime/stm32`
3. Moved the `src/runtime/stm32` to `src/runtime/crt/contrib/stm32`. I have chosen the `contrib' because it looks like other files in the directory are sort of common implementation, while the `stm32` is code-generator specific. If you prefer, not to use `contrib` for this - I will align the runtime and tests.
4. Moved the C part of our CI test to `tests/crt/contrib/stm32`
Now, I am looking at this comment:
> it seems like you could use the TVM C runtime's crt_backend_api.c and implement TVMPlatformMemoryAllocate(). Curious if that would be an option for you? btw, it would be totally fine with me if you wanted to split this file into a separate CRT library e.g. src/runtime/crt/backend to avoid including the rest of src/runtime/crt/common.
I have following observations:
1. Current TVM generates C code with these calls:
```
TVMAPISetLastError()
TVMGetLastError()
TVMBackendAllocWorkspace()
TVMBackendFreeWorkspace()
```
These would be good candidates for including with the `crt_backend_api.c`. Therefore, moving the `TVMAPISetLastError()` and the `TVMGetLastError()` there from the `crt_runtime_api.c`, and moving the `TVMBackendParallelLaunch()`, `TVMBackendRegisterSystemLibSymbol()` out to `crt_runtime_api.c`.
This way, the `crt_backend_api.c` would carry a minimal support required by the C target code generator.
2. Currently, the `TVMBackendAllocWorkspace` does some error handling. I would propose to leave the error handling to the caller, while the `TVMBackendAllocWorkspace` would just return `NULL` in case of error.
The reason is that the error handling and logging are platform-specific and even application-specific - eg. `malloc` does not do error handling, for example.
Handling this in the `TVMBackendAllocWorkspace` requires implementing the `TVMLogf` and the `TVMPlatformAbort`, which we will not be able to test. For example, in our `stm32` TVM test, I would have to implement the `x86` specific error handling, while for the Disco board I would set it to NOOP since the applications handles errors in its own way.
3. If we were to reduce the `crt_backend_api.c` to its minimal form as I have described above, perhaps it makes sense to have the application implement these few methods directly, i.e. no need to include the `tvm/crt/crt_backend_api.c` implementation, only the `crt_backend_api.h` header as interface ?
What do you think ?
Do not review the `tests/crt/contrib/stm32` yet - I will align it and clean after I have your opinion on the above.
cheers
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-807440103
@stoa
Hi Arthur! That would be great. Feel free to open a new topic categorized under Development > RFC on our [discuss forum](https://discuss.tvm.apache.org/). If you look in that [category](https://discuss.tvm.apache.org/c/development/rfc/8) you can see some examples of how people have titled and structured their RFCs. However, the process is fairly informal at this point. This PR would be a great PoC to backlink from the RFC as well, so people can read more specifics if they like.
Looking forward to it!
Thanks,
Andrew
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-855391490
@areusch @tqchen
Hello,
Let then sort out the review feedback in the order of importance:
> I'd like to avoid merging an alternative implementation of crt_backend_api.c if possible. Could we merge the other PR first and then use it? I'd like to avoid replacing layers below TVM's operator libraries where possible.
I can propose the following:
- Do we agree on the following "software stack" ?:
inference_application[c,so]
|
c_backend_api.[h,c]
| |
c_runtime_api[h,c' |
TVM RUNTIME AOT
| |
platform-specific[h,c]
- Do we agree that, in the AOT path, we do not want to include/compile the c_runtime_api.[h,c] ?
- Do we agree that the 'g_last_error' (TVMGetSetLastError()) is set by the running inference application and not by the lower runtime layers: c_runtime_api|platform_specific ?
In this case, I can move the 'g_last_error' (TVMGetSetLastError()) from the c_runtime_api[h,c] to the 'c_backend_api.[h,c]' and remove the duplicate 'g_last_error' functionality from the platform-specific implementation, 'runtime.c'.
The 'TVMAPIErrorf()' from the 'c_runtime_api.c' will have to be changed not to use the 'g_last_error', I have an impression that it is already pretty much the case.
> Platform-specific defines in 'ai_platform.h': I need to verify and will get back on this point quickly.
I am noticing that some of similar defines exist in 'c_runtime_api.h' and this is used pretty extensively throughout the TVM.
However, if we agree that the 'c_runtime-api[h,c]' is not to be used within the AOT path, we cannot use it.
I can propose to create some sort of 'c_runtime_defs.h' that would include platform-specific defines for:
- AI_API_ENTRY = TVM_DLL
- AI_API_ALIGNED
- AI_API_WEAK = TVM_WEAK
- etc.
The platform-specific implementation can include this file then instead of the 'c_runtime_api.h'.
Still the question of testing all different defines remain - but this is already the case with the 'c_runtime_api.h'.
> i think there are still some open items on the C code (e.g. the .clang-format should get removed and reformatted; may need to see if files in src/runtime/contrib/stm32 should be somehow placed under src/runtime/crt, src/runtime/micro, or src/micro, since they aren't intended to be compiled with the C++ runtime), but not sure if you're waiting to address the ai_platform.h question there.
Moving to a different code location is not an issue, please let us know which directory is most suitable.
And I also see that there is no reason for us to not reformat the C code - OK.
> I guess my concern with keeping the headers in place is that TVM code is supposed to be fairly uniform in style.
I will remove the headers.
> 'test_mnist_quant_fp': just wondering if the delete here was intentional?
Yes, intentional. We do not have a possibility to host these test-cases for downloading and they cannot be dowloaded from elsewhere.
Resuming: If we agree on re-organizing the 'c_runtime-api' as outlined above, I will align this PR and we will not need a separate PR in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r645850231
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -362,60 +377,18 @@ def check_result(tflite_results, tvm_results):
# ========================================================
-# test_mnist_quant_fp
Review comment:
just wondering if the delete here was intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-873449397
@areusch @tqchen
Hello Andrew
I have double-checked and the TVM does not seem to generate the TVMAPISetLastError with its operator code.
Therefore, you're right - this should not be blocking the STM32 port. We would use a STM32 own error reporting function and leave the c_runtime/backend_api untouched with this PR.
Later we will align with whatever outcome of the new RFC.
How about this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r626494235
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
How are you, Andrew @areusch
I have reorganized the code like so:
1. Removed the `apps/stm32`
2. Removed linker script code from the `emitter.py`
3. Removed the copy of `graph_runtime` from `src/runtime/stm32`
3. Moved the `src/runtime/stm32` to `src/runtime/crt/contrib/stm32`. I have chosen the `contrib' because it looks like other files in the directory are sort of common implementation, while the `stm32` is code-generator specific. If you prefer, not to use `contrib` for this - I will align the runtime and tests.
4. Moved the C part of our CI test to `tests/crt/contrib/stm32`
Now, I am looking at this comment:
> it seems like you could use the TVM C runtime's crt_backend_api.c and implement TVMPlatformMemoryAllocate(). Curious if that would be an option for you? btw, it would be totally fine with me if you wanted to split this file into a separate CRT library e.g. src/runtime/crt/backend to avoid including the rest of src/runtime/crt/common.
I have two observations:
1. Current TVM generates C code with these calls:
```
TVMAPISetLastError()
TVMGetLastError()
TVMBackendAllocWorkspace()
TVMBackendFreeWorkspace()
```
These would be good candidates for including with the `crt_backend_api.c`. Therefore, moving the `TVMAPISetLastError()` and the `TVMGetLastError()` there from the `crt_runtime_api.c`, and moving the `TVMBackendParallelLaunch()`, `TVMBackendRegisterSystemLibSymbol()` out to `crt_runtime_api.c`.
This way, the `crt_backend_api.c` would carry a minimal support required by the C target code generator.
2. Currently, the `TVMBackendAllocWorkspace` does some error handling. I would propose to leave the error handling to the caller, while the `TVMBackendAllocWorkspace` would just return `NULL` in case of error.
The reason is that the error handling and logging are platform-specific and even application-specific - eg. `malloc` does not do error handling, for example.
Handling this in the `TVMBackendAllocWorkspace` requires implementing the `TVMLogf` and the `TVMPlatformAbort`, which we will not be able to test. For example, in our `stm32` TVM test, I would have to implement the `x86` specific error handling, while for the Disco board I would set it to NOOP since the applications handles errors in its own way.
3. If we were to reduce the `crt_backend_api.c` to its minimal form as I have described above, perhaps it makes sense to have the application implement these few methods directly, i.e. no need to include the `tvm/crt/crt_backend_api.c` implementation, only the `crt_backend_api.h` header as interface ?
What do you think ?
Do not review the `tests/crt/contrib/stm32` yet - I will align it and clean after I have your opinion on the above.
cheers
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r715206463
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -27,7 +28,8 @@
# 1 = INFO messages are not printed
# 2 = INFO and WARNING messages are not printed
# 3 = INFO, WARNING, and ERROR messages are not printed
-os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
+if "TF_CPP_MIN_LOG_LEVEL" not in os.environ:
Review comment:
what's the reason for this change? at least could you put this in a `pytest` [fixture](https://docs.pytest.org/en/6.2.x/fixture.html) and put `os.environ` back to what it was before after the test finishes? generally, i don't think it's expected that a test case affects logging like this, though.
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -502,7 +397,7 @@ def __compute_data_placement(self):
#
tensor = self.__get_tensor_from_node(nid, 0)
- if self.inputs_static == 1:
+ if self.inputs_static == True:
Review comment:
note that python idiom is not to `== True` but merely to `if self.inputs_static:`
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ #
+ # Constants:
+ #
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ self.activations_static = include_activations
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def __tensor_is_output(self, nid, idx):
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def __get_tensor_from_node(self, nid, idx):
+ # 'eid' is index into the dltype', 'shape', etc.
+ eid = self.node_row_ptr_[nid] + idx
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
Review comment:
do you mind removing the lines with only `#` here and elsewhere? more commonly in TVM is
```
ndim = len(dims)
# Get tensor size
size = _get_tensor_size_bytes(dims, dltype)
```
also consider removing the comment when it doesn't add anything beyond the function name.
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
Review comment:
could you update the comment to indicate which option this is?
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ #
+ # Constants:
+ #
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ self.activations_static = include_activations
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
Review comment:
this is a bit of a nit, but in general elsewhere in `python/` we prepend the `_` for private members e.g. `self._nodes`. do you mind updating this?
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -1449,63 +1389,81 @@ def __emit_run(self, name, out_h, out_c):
out_c.write(f"\n")
out_c.write(f" fclose(DumpFile_p); \n")
out_c.write(f"#endif \n")
- out_c.write(f"\n")
-
- out_c.write(f" return AI_STATUS_OK; \n")
- out_c.write(f"}} \n")
- out_c.write(f"\n")
- # ==========================================================
- # __emit_create_destroy
- # ==========================================================
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
def __emit_create_destroy(self, name, out_h, out_c):
Review comment:
sorry one other style point. in Python `__` (double-underscore or "dunder") methods are considered reserved for use by low-level Python classes. while these aren't quite exactly proper dunder (dunder methods both begin and end with `__`, do you mind using the more common single-underscore prefix for these helper functions?
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
Review comment:
i think you could just simplify this function to: `return dst + src`
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ #
+ # Constants:
+ #
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ self.activations_static = include_activations
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def __tensor_is_output(self, nid, idx):
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def __get_tensor_from_node(self, nid, idx):
+ # 'eid' is index into the dltype', 'shape', etc.
+ eid = self.node_row_ptr_[nid] + idx
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size_bytes(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+ buffer_list_ = {}
+
+ nid = 0
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self.__tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+ for output in self.outputs_:
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
Review comment:
from other file: i would suggest not extracting these from self.graph_ and instead just accessing them using self.graph_["nodes"]. it may be very slightly slower, but it makes the code easier to maintain as you don't need to keep track of where each of these properties comes from. if you want to assert there aren't unknown keys in the graph, you could assert(list(sorted(self.graph_)) == ["arg_nodes", "attrs", "heads", "metadata", "node_row_ptr", "nodes"])
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ #
+ # Constants:
+ #
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ self.activations_static = include_activations
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def __tensor_is_output(self, nid, idx):
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def __get_tensor_from_node(self, nid, idx):
+ # 'eid' is index into the dltype', 'shape', etc.
+ eid = self.node_row_ptr_[nid] + idx
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size_bytes(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+ buffer_list_ = {}
+
+ nid = 0
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self.__tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+ for output in self.outputs_:
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ elif key == "metadata":
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The ModuleLibraryFormat object to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "executor-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM Runtime Module
+ The module to parse.
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ #
+ # Sort params for debugging
+ #
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self.params_ = params_dict
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ def __emit_params_data(self, name, out_h, out_c):
+ """ Emits the network_data[c,h] files with parameters."""
+
+ name_upper = name.upper()
+
+ #
+ # XXX_data.h
+ #
+ out_h.write(
+ textwrap.dedent(
+ f"""\
+ #ifndef __{name_upper}_DATA_H_
+ #define __{name_upper}_DATA_H_
+
+ #include \"ai_runtime_api.h\"
+
+ AI_API_ENTRY
+ const ai_ptr ai_{name}_data_weights_get (void);
+
+ #endif /* __{name_upper}_DATA_H_ */
+ """
+ )
+ )
+
+ #
+ # XXX_data.cc
+ #
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ #include \"{name}_data.h\"
+
+ const ai_ptr ai_{name}_data_weights_get (void)
+ {{
+ AI_ALIGNED({self.DATA_ALIGNMENT_BYTES}) static const __attribute__ ((section(\".nn_weights\"))) uint8_t s_{name}_weights[] = {{
+ """
+ )
+ )
+
+ #
+ # Weights are arranged in the order of 'params_'
+ #
+ offset = 0
+
+ for key in self.params_:
+ data = self.params_[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ #
+ # Align by emitting garbage between un-aligned data
+ #
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.weights_[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ }};
+ return (const ai_ptr)s_{name}_weights;
+ }}
+ """
+ )
+ )
+
+ def __emit_open(self, name, out_h, out_c):
+ """Emits the network.h file with a few network defines and
+ writes the header part of the network.c file."""
+
+ name_upper = name.upper()
+
+ input_size = len(self.input_data_)
+ output_size = len(self.output_data_)
+
+ #
+ # XXX.h
+ #
+ out_h.write(
+ textwrap.dedent(
+ f"""\
+ #ifndef __AI_{name_upper}_H__
+ #define __AI_{name_upper}_H__
+
+ #include \"ai_runtime_api.h\"
+
+ #define _{name_upper}_INPUTS_COUNT_ ({input_size})
+ #define _{name_upper}_OUTPUTS_COUNT_ ({output_size})
+ #define _{name_upper}_ACTIVATION_BYTES_ ({self.activations_size_})
+ """
+ )
+ )
+
+ #
+ # XXX.c
+ #
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ #include <stdio.h>
+
+ #include \"dlpack/dlpack.h\"
+ #include \"tvm/runtime/c_runtime_api.h\"
+ #include \"{name}.h\"
+ #include \"{name}_data.h\"
+ """
+ )
+ )
+
+ def __emit_close(self, name, out_h, out_c):
+ """ Emits the ai_model_info structure. """
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+
+ #
+ # XXX.h
+ #
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ #
+ # XXX.c
+ #
+
+ if self.activations_static == True:
+ out_c.write(
+ f'AI_ALIGNED({self.DATA_ALIGNMENT_BYTES}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self.activations_size_}];\n'
+ )
+ else:
+ out_c.write(f"AI_STATIC ai_ptr {name}_activations = NULL;")
+
+ #
+ # Emit network structure
+ #
+ num_inputs = len(self.input_data_)
+ num_outputs = len(self.output_data_)
+
+ tool_version = tvm.__version__
+ api_version = f"{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0"
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{
+ .name = \"{name}\",
+ .datetime = \"{dt_string}\",
+ .revision = \"{AI_TOOLS_REVISION}\",
+ .tool_version = \"{tool_version}\",
+ .api_version = \"{api_version}\",
+ .n_nodes = {self.nodes_size_},
+ .n_inputs = {num_inputs},
+ .n_outputs = {num_outputs},
+ .activations_size = {self.activations_size_},
+ .params_size = {self.weights_size_},
+ .activations = {name}_activations,
+ .inputs = _InputsList,
+ .outputs = _OutputsList,
+ .ai_get_params = &ai_{name}_data_weights_get,
+ .ai_create = &ai_{name}_create,
+ .ai_destroy = &ai_{name}_destroy,
+ .ai_run = &ai_{name}_run
+ }};
+ """
+ )
+ )
+
+ def __emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ def __emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self.quantization_:
+ quantization = self.quantization_[dl_tensor_name]
+ #
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ #
+ elif dl_tensor_name in self.output_data_ and "output" in self.quantization_.keys():
+ quantization = self.quantization_["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+
+ #
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ #
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};")
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ AI_STATIC ai_quantization_info {quant_name} = {{
+ .scale = {quant_name}_scale,
+ .zero_point = {quant_name}_zero_point,
+ .dim = -1
+ }};
+ """
+ )
+ )
+
+ return quant_name
+
+ return None
+
+ def __emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+ """ Emits the tensor instantiation code. """
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ byte_offset = tensor["byte_offset"]
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self.__emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ #
+ # Quantization
+ #
+ quant_name = self.__emit_tensor_quant(dl_tensor_name, out_c)
+
+ #
+ # Contents
+ #
+ # TODO: use the 'storage_id':
+ # " .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1])
+ #
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ AI_ALIGNED({self.DATA_ALIGNMENT_BYTES}) AI_STATIC ai_tensor {dl_tensor_name} = {{
+ .dltensor = {{
+ .data = (ai_ptr)(NULL),
+ .device = {{kDLCPU,0}},
+ .ndim = {ndim},
+ .dtype = {{{dtype}}},
+ .shape = {dl_tensor_name}_shape,
+ .strides = {dl_tensor_name}_strides,
+ .byte_offset = {byte_offset}
+ }},
+ """
+ )
+ )
+
+ #
+ # Figure out quantization, if exists
+ #
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ def __emit_activation_buffers(self, name, out_c):
+ # pylint: disable=unused-argument
+ """ Emits activation tensors, including inputs/outputs."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ //
+ // Inputs:
+ //
+ """
+ )
+ )
+ #
+ # shape/buffer
+ #
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ #
+ # tensor
+ #
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self.input_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ //
+ // Activations:
+ //
+ """
+ )
+ )
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ #
+ # Outputs:
+ #
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ //
+ // Outputs:
+ //
+ """
+ )
+ )
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self.output_data_:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ def __emit_params_buffers(self, name, out_c):
+ """ Emits all parameter tensors."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ //
+ // Weights: {name}
+ //
+ """
+ )
+ )
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ self.__emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ def __emit_network(self, name, out_c):
+ """ Emits prototypes for the network operator functions."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ //
+ // Network: {name}
+ //
+ """
+ )
+ )
+ for node in self.nodes_:
+ if node["op"] == "null":
+ continue
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ if func_name == "__nop":
+ continue
+
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ def __emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(
+ textwrap.indent(
+ textwrap.dedent(
+ f"""
+ //
+ // {dl_tensor_name}: storage_id:{storage_id}
+ //
+ {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset});
+ """
+ ),
+ " ",
+ )
+ )
+
+ def __emit_activation_init(self, name, out_c):
+ """ Emits buffer initialization code for activation tensors."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // {name}_configure_activations
+ // {DBAR}
+ AI_STATIC AI_INLINE
+ ai_status {name}_configure_activations (
+ const ai_ptr activations
+ )
+ {{
+ if (activations == NULL) {{
+ TVMAPISetLastError (\"Non-null activations arena is required for this model.\");
+ return AI_STATUS_ERROR;
+ }}
+ """
+ )
+ )
+
+ #
+ # Allocate inputs with the static model
+ #
+ if self.inputs_static == True:
+ for dl_tensor_name in self.input_data_:
+ tensor = self.input_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Prepare activation buffers
+ #
+ for dl_tensor_name in self.activations_:
+ tensor = self.activations_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ #
+ # Allocate outputs with the static model
+ #
+ if self.outputs_static == True:
+ for dl_tensor_name in self.output_data_:
+ tensor = self.output_data_[dl_tensor_name]
+ self.__emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
+
+ def __emit_params_init(self, name, out_c):
+ """ Emits buffer initialization code for params tensors."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // {name}_configure_weights
+ // {DBAR}
+ AI_STATIC AI_INLINE
+ ai_status {name}_configure_weights (
+ const ai_ptr weights
+ )
+ {{
+ if (weights == NULL) {{
+ TVMAPISetLastError(\"Non-null weights arena is required for this model.\");
+ return AI_STATUS_ERROR;
+ }}
+ """
+ )
+ )
+
+ for dl_tensor_name in self.weights_:
+ tensor = self.weights_[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(
+ textwrap.indent(
+ textwrap.dedent(
+ f"""\
+ //
+ // {dl_tensor_name}
+ //
+ {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset});
+ """
+ ),
+ " ",
+ )
+ )
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
+
+ def __emit_init(self, name, out_c):
+ """ Emits buffer initialization code."""
+
+ #
+ # {name}_configure_activations
+ #
+ self.__emit_activation_init(name, out_c)
+ #
+ # {name}_configure_weights
+ #
+ self.__emit_params_init(name, out_c)
+
+ def __emit_run(self, name, out_h, out_c):
+ """ Emits the run function code."""
+
+ out_h.write(
+ textwrap.dedent(
+ f"""
+ AI_API_ENTRY
+ ai_status ai_{name}_run (
+ ai_tensor *inputs[],
+ ai_tensor *outputs[]
+ );
+ """
+ )
+ )
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // ai_{name}_run
+ // {DBAR}
+ AI_API_ENTRY
+ ai_status ai_{name}_run (
+ ai_tensor *inputs[],
+ ai_tensor *outputs[]
+ )
+ {{
+ """
+ )
+ )
+
+ out_c.write(f"#if defined(_DUMP_INPUTS_) ")
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+ if node["op"] != "null":
+ out_c.write(f"|| defined(_DUMP_{node_name_upper}_) ")
+ out_c.write(f"\n")
+ out_c.write(f' FILE * DumpFile_p = fopen("dump.txt", "w"); \n')
+ out_c.write(f"#endif \n")
+ out_c.write(f"\n")
+
+ #
+ # Execute nodes one by one
+ #
+ nid = 0
+
+ for node in self.nodes_:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ if func_name == "__nop":
+ continue
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ #
+ # Prepare TVM packed function - this is the one called
+ #
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ #
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+ #
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for arg in node["inputs"]:
+ dl_tensor_name = self.__get_node_arg_name(arg)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self.weights_ and dl_tensor_name not in self.activations_:
+
+ assert dl_tensor_name in self.input_data_, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self.input_data_:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ if dl_tensor_name in self.weights_:
+ out_c.write(f"#ifdef _DUMP_INPUTS_ \n")
+ out_c.write(
+ f' TVMDumpBuffer ("{dl_tensor_name}", &{dl_tensor_name}, DumpFile_p); \n'
Review comment:
i see this function referenced here, but where is it implemented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-867774013
@areusch
Hello, Andrew
I do not understand the build failure. Who can explain it what the problem is ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-823447885
@areusch
Hello Andrew
Now back to your main comment.
There are indeed the board/X-Cube project specific files. They are in apps/stm32/ project directory, which is the X-Cube derived directory. Are you thinking of another place or renaming and adding the STM X-Cube to the name ?
There are also a few .py files in the python/tvm/contrib/stm32 directory that allow connecting to STM32 via the ST Link, which indeed use the pyserial package. I thought this functionality is generic enough and interesting enough for the STM32 users to put it together with the code emitter. If you feel these files belong to the X-Cube project, I can move them to the apps/stm32 or whichever place you'd suggest.
> here I think I see tests (in tests/micro) for the logic in python/tvm/contrib/stm32 and some of src/runtime/stm32
> right now our test strategy for TVM CI is to test only with virtualization (e.g. qemu or other emulators) on publicly available cloud hardware. so i think it would be hard to test logic such as the usb validation driver inside the TVM CI.
This micro/stm32 test verifies only the stm32/code emitter and the runtime/stm32,. It does not test the STLink or X-Cube project logic - no board necessary. It runs in the x86 standard environment.
> finally, I should say that out of necessity we do have some of this with microTVM (some RPC layers and pyserial code), but these are effectively ports of the official TVM RPC protocol and we test as much as possible against a virtual e.g. QEMU as well as have a defined path to testing with real hardware. what i don't know is whether e.g. the board-specific logic lives in another repo--if so, i wonder whether it would be easier to test that code in that repository.
There is no problem to moving the ST-Link logic to the X-Cube project apps/stm32 directory.
The application there is not run with the CI tests because a board would be necessary, this application is there as a demo for STM32 developers on how to use TVM with the HW board.
So, I would follow your lead here - please suggest how I could re-arrange the apps/stm32 and contrib/stm32 files to be more in line with the micro.
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r713474347
##########
File path: src/runtime/crt/contrib/stm32/runtime.c
##########
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file runtime.c
+ * \brief A minimal "C" runtime support required by the TVM
+ * generated C code. Declared in "runtime/c_backend_api.h"
+ * and "runtime/c_runtime_api.h"
+ */
+
+#include <assert.h>
+#include <malloc.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <tvm/runtime/c_backend_api.h>
+#include <tvm/runtime/crt/error_codes.h>
+
+static char* g_last_error = NULL;
+
+// ====================================================
+// TVMPlatformMemoryAllocate
+// ====================================================
+tvm_crt_error_t TVMPlatformMemoryAllocate(size_t num_bytes, DLDevice dev, void** out_ptr) {
Review comment:
will you guys also provide timing functions? can be in a follow-on
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1618 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import shutil
+import tarfile
+import re
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+# ==========================================================
+# _fix_name
+# ==========================================================
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==========================================================
+ # _get_node_arg_name
+ # ==========================================================
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
Review comment:
can you remove the `#` spacer lines above and below these comments, here and elsewhere? it will match the rest of the TVM python code style better.
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1643 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
this file is a duplicate of `python/tvm/contrib/stm32/emitter.py`. can you remove one of them? also please address the outstanding comments on the other one, whichever you choose. there are a few areas that could be improved to make the code more maintainable.
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
Review comment:
ping on this one
##########
File path: src/runtime/crt/contrib/stm32/ai_runtime_api.c
##########
@@ -0,0 +1,402 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file ai_runtime_api.c
+ * \brief The runtime API for the TVM generated C code.
+ */
+
+// LINT_C_FILE
+
+#include "ai_runtime_api.h"
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+// =======================================================
+// ai_network_t
+// =======================================================
+
+typedef struct {
+ ai_model_info* info;
+ ai_tensor** inputs;
+ ai_tensor** outputs;
+ ai_ptr activations;
+ const char* error;
+} ai_network_t;
+
+//
+// .nn_models_info section
+//
+extern uint32_t __models_section_start__;
Review comment:
nit: should this be uintptr_t?
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1618 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import shutil
+import tarfile
+import re
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+# ==========================================================
+# _fix_name
+# ==========================================================
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
Review comment:
could you simplify this and the below logic and use boolean in place of 1/0?
self.inputs_static = bool(include_inputs)
self.outputs_static = bool(include_outputs)
assert self.inputs_static == self.outputs_static, "Static inputs are not allowed without activations."
##########
File path: tests/crt/contrib/stm32/src/main.c
##########
@@ -0,0 +1,461 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <inttypes.h>
+#include <math.h>
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "ai_runtime_api.h"
+#include "network.h"
+#include "network_data.h"
+
+//
+// Network that we are testing
+//
+extern ai_model_info network_network;
+
+//
+// Dummy: for the runtime
+//
+uint32_t __models_section_start__ = (uint32_t)&network_network;
+uint32_t __models_section_end__ = (uint32_t)&network_network + sizeof(ai_model_info);
+
+static ai_model_info* _model_p = &network_network;
+
+//
+// Global handle to reference the instantiated NN
+//
+static ai_handle _network = AI_HANDLE_NULL;
+
+static uint8_t LoadInputImg(const char* filename, ai_tensor* input);
+static int32_t quantize_val(float val, ai_quantization_info* quant);
+static float dequantize_val(int32_t val, ai_quantization_info* quant);
+
+// =================================================================
+// Convert_Fixed_To_Float
+// =================================================================
+static float Convert_Fixed_To_Float(uint8_t data, int8_t fl) {
+ uint8_t val = data;
+ float x;
+ if (fl >= 0) {
+ x = ((float)val) / (float)(1 << fl); // NOLINT
+ } else {
+ x = ((float)val) / (float)(1 >> fl); // NOLINT
Review comment:
i don't think this will work--(1 >> nonzero) is always going to be 0.
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1665 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import os
+
+# import sys
+import shutil
+import tarfile
+import re
+import json
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "============================================================"
+
+# ==========================================================
+# _fix_name
+#
+# Replace ':' with '_' in names like 'InputImg:0'
+# ==========================================================
+
+
+def _fix_name(node_name):
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
Review comment:
are you guys ok with deleting the `# =======` comments?
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1618 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import shutil
+import tarfile
+import re
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+# ==========================================================
+# _fix_name
+# ==========================================================
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==========================================================
+ # _get_node_arg_name
+ # ==========================================================
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self.__tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ elif key == "metadata":
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
Review comment:
can you further specify the type? i think this is GarphExecutorFactory.
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1618 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import shutil
+import tarfile
+import re
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+# ==========================================================
+# _fix_name
+# ==========================================================
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==========================================================
+ # _get_node_arg_name
+ # ==========================================================
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self.__tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
Review comment:
i would suggest not extracting these from self.graph_ and instead just accessing them using self.graph_["nodes"]. it may be very slightly slower, but it makes the code easier to maintain as you don't need to keep track of where each of these properties comes from. if you want to assert there aren't unknown keys in the graph, you could `assert(list(sorted(self.graph_)) == ["arg_nodes", "attrs", "heads", "metadata", "node_row_ptr", "nodes"])`
##########
File path: include/tvm/runtime/c_runtime_api.h
##########
@@ -190,6 +190,7 @@ TVM_DLL void TVMAPISetLastError(const char* msg);
* \return error info
*/
TVM_DLL const char* TVMGetLastError(void);
+
Review comment:
ping on this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r713992109
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1643 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
Removed the python/tvm/contrib/stm32.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] tqchen commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
tqchen commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-807048532
Thanks @stoa , given this is a new feature. It would be great if you can post an RFC thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r616853488
##########
File path: apps/stm32/.clang-format
##########
@@ -0,0 +1,2 @@
+DisableFormat: true
Review comment:
There are some project generated files that I do not feel like changing.
The DisableFormat removes the include order errors, which somehow are not siabled even by disabling the formatting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-807144342
hi @stoa, thanks for your contribution! I agree with @tqchen an RFC would be great.
looking over the code I see a lot of concepts implemented that are fairly similar to what we have implemented in microTVM now:
- a remote execution driver
- a project generation script
- a minimal C runtime, including graph runtime and c_backend_api.h implementation
For microTVM, we're taking the approach of reusing C runtime components between platforms and/or SoCs, and allowing platform-specific e.g. memory allocators by implementing `include/tvm/runtime/crt/platform.h`. We are also adopting a standard "interface layer" between TVM and generated firmware code (composed of "Model Library Format," "Project API," and the "TVM RPC server"), which provides a great way to reduce boilerplate code when targeting µC.
It would be great if we could consider a way to unify your approach with our microTVM approach. It seems like the STM32 firmware code could become an implementation of our Project API (similar to the [host](https://github.com/areusch/incubator-tvm/tree/project-generator/src/runtime/crt/host) demo project). This would allow us to bring further improvements to TVM currently under development (e.g. AOT runtime, graph-level optimization, `tvmc` integration), as well as enable AutoTVM on your platform.
Here are some pointers to RFCs that describe our approach in more detail:
- [microTVM M2 roadmap](https://discuss.tvm.apache.org/t/tvm-microtvm-m2-roadmap/8821)
- [Model Library Format RFC](https://discuss.tvm.apache.org/t/rfc-tvm-model-library-format/9121), now committed
- [Project Generator API RFC](https://discuss.tvm.apache.org/t/rfc-tvm-project-api/9449), PoC [here](https://github.com/areusch/incubator-tvm/tree/project-generator).
- [AOT Runtime RFC](https://discuss.tvm.apache.org/t/implementing-aot-in-tvm/9206)
Please also feel free to suggest improvements or changes to our current implementation. It is very much in development and we would like to encourage contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-922745127
@areusch
How are you, Andrew
Please review the most recent commit.
cheers
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch merged pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch merged pull request #7742:
URL: https://github.com/apache/tvm/pull/7742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa edited a comment on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa edited a comment on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-855391490
@areusch @tqchen
Hello,
Let then sort out the review feedback in the order of importance:
> I'd like to avoid merging an alternative implementation of crt_backend_api.c if possible. Could we merge the other PR first and then use it? I'd like to avoid replacing layers below TVM's operator libraries where possible.
I can propose the following:
- Do we agree on the following "software stack" ?:
```
inference_application[c,so]
|
c_backend_api.[h,c]
| |
c_runtime_api[h,c' |
TVM RUNTIME AOT
| |
platform-specific[h,c]
```
- Do we agree that, in the AOT path, we do not want to include/compile the c_runtime_api.[h,c] ?
- Do we agree that the 'g_last_error' (TVMGetSetLastError()) is set by the running inference application and not by the lower runtime layers: c_runtime_api|platform_specific ?
In this case, I can move the 'g_last_error' (TVMGetSetLastError()) from the c_runtime_api[h,c] to the 'c_backend_api.[h,c]' and remove the duplicate 'g_last_error' functionality from the platform-specific implementation, 'runtime.c'.
The 'TVMAPIErrorf()' from the 'c_runtime_api.c' will have to be changed not to use the 'g_last_error', I have an impression that it is already pretty much the case.
> Platform-specific defines in 'ai_platform.h': I need to verify and will get back on this point quickly.
I am noticing that some of similar defines exist in 'c_runtime_api.h' and this is used pretty extensively throughout the TVM.
However, if we agree that the 'c_runtime-api[h,c]' is not to be used within the AOT path, we cannot use it.
I can propose to create some sort of 'c_runtime_defs.h' that would include platform-specific defines for:
- AI_API_ENTRY = TVM_DLL
- AI_API_ALIGNED
- AI_API_WEAK = TVM_WEAK
- etc.
The platform-specific implementation can include this file then instead of the 'c_runtime_api.h'.
Still the question of testing all different defines remain - but this is already the case with the 'c_runtime_api.h'.
> i think there are still some open items on the C code (e.g. the .clang-format should get removed and reformatted; may need to see if files in src/runtime/contrib/stm32 should be somehow placed under src/runtime/crt, src/runtime/micro, or src/micro, since they aren't intended to be compiled with the C++ runtime), but not sure if you're waiting to address the ai_platform.h question there.
Moving to a different code location is not an issue, please let us know which directory is most suitable.
And I also see that there is no reason for us to not reformat the C code - OK.
> I guess my concern with keeping the headers in place is that TVM code is supposed to be fairly uniform in style.
I will remove the headers.
> 'test_mnist_quant_fp': just wondering if the delete here was intentional?
Yes, intentional. We do not have a possibility to host these test-cases for downloading and they cannot be dowloaded from elsewhere.
Resuming: If we agree on re-organizing the 'c_runtime-api' as outlined above, I will align this PR and we will not need a separate PR in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-807374458
Hello @areusch and @tqchen .
Thanks for the prompt consideration. We are very interested in collaborating in the mTVM project and to contribute to the TVM model targeting micro/bare-metal devices.
In practical terms, I will post an RFC like you suggested, and then we can discuss on the basis of the mTVM RFC and the new one. Could you point me at how one starts a TVM RFC thread ?
Thanks and cheers,
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r713992580
##########
File path: python/tvm/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1618 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import shutil
+import tarfile
+import re
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_TOOLS_VERSION_MAJOR = 1
+AI_TOOLS_VERSION_MINOR = 8
+AI_TOOLS_VERSION_MICRO = 0
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+# ==========================================================
+# _fix_name
+# ==========================================================
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+# ==========================================================
+# get_input_tensor_name
+# ==========================================================
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+# ==========================================================
+# get_output_tensor_name
+# ==========================================================
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+# ==========================================================
+# _get_node_args_name
+# ==========================================================
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+# ==========================================================
+# _get_node_arg_types_name
+# ==========================================================
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+# ==========================================================
+# _get_type_size
+# ==========================================================
+
+
+def _get_type_size(dltype):
+
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_type_data
+# ==========================================================
+
+
+def _get_type_data(dltype):
+
+ if dltype == "uint64":
+ return "kDLUInt, 64, 1"
+ if dltype == "int64":
+ return "kDLInt, 64, 1"
+ if dltype == "float32":
+ return "kDLFloat, 32, 1"
+ if dltype == "uint32":
+ return "kDLUInt, 32, 1"
+ if dltype == "int32":
+ return "kDLInt, 32, 1"
+ if dltype == "uint16":
+ return "kDLUInt, 16, 1"
+ if dltype == "int16":
+ return "kDLInt, 16, 1"
+ if dltype == "uint8":
+ return "kDLUInt, 8, 1"
+ if dltype == "int8":
+ return "kDLInt, 8, 1"
+
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+# ==========================================================
+# _get_aligned_offset
+# ==========================================================
+
+
+def _get_aligned_offset(offset, dltype):
+
+ align = _get_type_size(dltype)
+
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+
+ return offset
+
+
+# ==========================================================
+# _get_tensor_elts
+# ==========================================================
+
+
+def _get_tensor_elts(dims):
+ size = 1
+ for dim in dims:
+ size = size * dim
+ return size
+
+
+# ==========================================================
+# _get_tensor_size
+# ==========================================================
+
+
+def _get_tensor_size(dims, dltype):
+ size = _get_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+# ==================================================================
+# _preprocess_code
+# ==================================================================
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ dst = "#include <stdio.h>\n" "#include <math.h>\n" '#include "stm32lib.h"\n\n' ""
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+# ==========================================================
+# CodeEmitter
+# ==========================================================
+
+
+class CodeEmitter(object):
+ """Code emitter class/utility."""
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Constants:
+ #
+ self.data_alignment = 8
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ if include_activations is True:
+ self.activations_static = 1
+ else:
+ self.activations_static = 0
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = 1
+ else:
+ self.inputs_static = 0
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == 1
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = 1
+ else:
+ self.outputs_static = 0
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ # ==================================================================
+ # __extract_quantization_info
+ # ==================================================================
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ # ==========================================================
+ # _get_node_arg_name
+ # ==========================================================
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ # ==========================================================
+ # __tensor_is_output
+ # ==========================================================
+
+ def __tensor_is_output(self, nid, idx):
+
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ # ==========================================================
+ # __get_tensor_from_node
+ # ==========================================================
+
+ def __get_tensor_from_node(self, nid, idx):
+ #
+ # 'eid' is index into the dltype', 'shape', etc.
+ #
+ eid = self.node_row_ptr_[nid] + idx
+
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ # ==================================================================
+ # __compute_data_placement
+ # ==================================================================
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+
+ buffer_list_ = {}
+
+ nid = 0
+
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self.__tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+
+ for output in self.outputs_:
+
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == 1:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ # ==========================================================
+ # _parse_model
+ # ==========================================================
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self.nodes_ = self.graph_["nodes"]
+ elif key == "arg_nodes":
+ self.arg_nodes_ = self.graph_["arg_nodes"]
+ elif key == "node_row_ptr":
+ self.node_row_ptr_ = self.graph_["node_row_ptr"]
+ elif key == "heads":
+ self.outputs_ = self.graph_["heads"]
+ elif key == "attrs":
+ self.attrs_ = self.graph_["attrs"]
+ elif key == "metadata":
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ #
+ # Build all tensor lists
+ #
+ self.__compute_data_placement()
+
+ #
+ # Extract quantization info for inputs/outputs
+ #
+ if quantization is not None:
+ self.__extract_quantization_info(quantization)
+
+ # ==========================================================
+ # parse_library_format
+ # ==========================================================
+
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ #
+ # Extract informations from the Model Library Format
+ #
+ graph_file = os.path.join(extract_path, "runtime-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+ #
+ # Map -> Python Dict
+ #
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ #
+ # Sort params for debugging
+ #
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+
+ self.graph_ = graph_dict
+ self.params_ = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ # ==========================================================
+ # parse_module
+ # ==========================================================
+
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module
Review comment:
I kept 2 versions. Docstring fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-830804983
Hello, Andrew @areusch
> 1. I think all the code currently in apps/stm32 should live outside the TVM tree.
This is acceptable - I can remove it from the TVM.
> 2. I think we should move the code in src/runtime/stm32 plus the code in tests/micro/stm32/src/ into apps/micro/stm32. Organizationally, I'd prefer to keep src/runtime dedicated to the TVM runtime logic. Since, as discussed on the RFC, the medium-term direction will be to move the TVM runtime in the direction of the one you proposed, I would rather consider this an "app" than an official implementation. As we move the TVM runtime forward, presumably this will eventually resemble a small shim or would disappear if the TVM runtime APIs are moved to match these (I would sort of expect there to be some minor differences in the long run between TVM and the cross-toolchain STM X-Cube AI APIs).
If we do this, it will complicate our own code maintenance and we may overtime diverge from TVM. This is why:
1. The src/runtime and src/runtime/contrib already contain a number of target-specific runtime implementations, while the apps contains really the application/integration examples.
2. As I have mentioned in the RFC, we insist on separating the Application code from the Tools. The idea for us is to have our code generator + runtime part of the TVM distributed with the TVM. We want to avoid to refer to tvm/apps for accessing the runtime layer. If this were to be checked into the apps directory, we will certainly have to structure it differently with our tools offering, which will lead to needing to maintain two repositories - not what we need. **For us it is an official TVM for STM32 implementation rather than an app**.
3. I sort of disagree that the slim layer is bound to completely disappear. On the contrary, there will most certainly be a need to such a slim adaptation layer between the micro TVM API and any particular target system, for example for the memory allocation part. Isn't it better that these slim layers live in the src/runtime rather then in apps ?
Perhaps we can figure out some middle ground here:
- We could move the src/runtime/stm32 to the src/runtime/contrib/stm32, or
- We could rename it into src/runtime/(contrib)/x-cube to resemble more coreml, rocm, cuda, etc.
- We do not have any particular position on the tests/ directory - I will integrate our test anyway you deem appropriate, can go to the apps/stm32. However, I did not understand: are you proposing to separate the src part of the test from the rest ?
> I'd suggest we don't output device-specific things (e.g. linker scripts) in `python/tvm/contrib/stm32/emitter.py.`
I will remove this.
The following is already taken care of:
> everything in the TVM tree needs to get apache-licensed, so by checking in that code we need to be sure that's cool with you guys
The TVM license header is already added to all code.
> anything in the TVM tree will need a defined path to be tested (ideally one that's automated) and any maintenance will need to go through our code review process
This is understood. This is why we want our runtime layer to be integrated as a "real" runtime with the TVM - it will be the only place it lives in and is maintained: it is TVM specific..
What do you think ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r625345084
##########
File path: src/runtime/stm32/runtime.c
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*!
+ * \file runtime.c
+ * \brief A minimal "C" runtime support required by the TVM
+ * generated C code. Declared in "runtime/c_backend_api.h"
+ * and "runtime/c_runtime_api.h"
+ */
+
+#include <stdio.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <string.h>
+#include <malloc.h>
+
+#include "tvm/runtime/c_runtime_api.h"
+
+static char * g_last_error = NULL;
+
+// ====================================================
+// TVMBackendAllocWorkspace
+// ====================================================
+void *
+TVMBackendAllocWorkspace(
Review comment:
it seems like you could use the TVM C runtime's [`crt_backend_api.c`](https://github.com/apache/tvm/blob/main/src/runtime/crt/common/crt_backend_api.c#L35) and implement `TVMPlatformMemoryAllocate()`. Curious if that would be an option for you? btw, it would be totally fine with me if you wanted to split this file into a separate CRT library e.g. `src/runtime/crt/backend` to avoid including the rest of `src/runtime/crt/common`.
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
this looks like a copy of `src/runtime/graph_executor/graph_executor.cc`. I'm curious why you prefer to use this one rather than the C-runtime `src/runtime/crt/graph_executor`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-869893211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r720852569
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1369 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
Review comment:
just curious: in general i think our code emitter should properly include all the stdlibs required to compile the code. i think maybe originally this included `"ai_platform_api.h"` if my memory serves? just curious if you need this function still?
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -0,0 +1,390 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import shutil
+import struct
+import sys
+
+import numpy as np
+
+import tensorflow as tf
+
+import tvm
+import tvm.relay as relay
+from tvm.micro.contrib import stm32
+from tvm.contrib.download import download
+from tvm import testing
+
+import conftest
+
+NUM_ITERATIONS = 10
+
+BUILD_DIR = "build"
+
+# =========================================================
+# get_data
+# =========================================================
+def get_data(in_data_shapes, in_data_dtypes):
+ """Generate a uint8 image."""
+ assert len(in_data_shapes) == 1, "Only single input models are supported."
+ in_data = OrderedDict()
+ for shape_name, shape in in_data_shapes.items():
+ for dtype_name, dtype in in_data_dtypes.items():
+ if dtype_name == shape_name:
+ in_data[shape_name] = np.random.uniform(size=shape).astype(dtype)
+ in_data = np.random.uniform(size=shape).astype("uint8")
+ break
+ if shape_name not in in_data.keys():
+ raise ValueError("Shape and dtype dictionaries do not fit.")
+
+ return in_data
+
+
+# ==================================================================
+# dump_image
+# ==================================================================
+def dump_image(filename, image):
+ # Flatten image
+ image_data = image.flatten()
+ outputRaw = []
+ # Raw binary format
+ for i in range(0, len(image_data)):
+ outputRaw.append(struct.pack("<B", int(image_data[i]) & 0xFF))
+
+ # Dump image in raw binary format
+ f = open(filename, "wb")
+ for i in range(0, len(outputRaw)):
+ f.write(outputRaw[i])
+ f.close()
+
+
+# ==================================================================
+# scale_input_data
+# ==================================================================
+def scale_input_data(input_details, data):
+ if input_details["dtype"] == np.uint8 or input_details["dtype"] == np.int8:
+ input_scale, input_zero_point = input_details["quantization"]
+ print(
+ "== TFLite input quantization: scale={}, zero={}".format(input_scale, input_zero_point)
+ )
+ data = data / input_scale + input_zero_point
+ data = data.astype(input_details["dtype"])
+ return data
+
+
+# ==================================================================
+# scale_output_data
+# ==================================================================
+def scale_output_data(output_details, data):
+ if output_details["dtype"] == np.uint8 or output_details["dtype"] == np.int8:
+ output_scale, output_zero_point = output_details["quantization"]
+ print(
+ "== TFLite output quantization: scale={}, zero={}".format(
+ output_scale, output_zero_point
+ )
+ )
+ data = data.astype(np.float32)
+ data = (data - output_zero_point) * output_scale
+ return data
+
+
+# ========================================================
+# get_tflite_model
+# ========================================================
+def get_tflite_model(model_path):
+
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()
+ output_details = interpreter.get_output_details()
+
+ #
+ # Figure out shapes and
+ #
+
+ shape_dict = {}
+ dtype_dict = {}
+
+ for input in input_details:
+ input_name = input["name"]
+ input_shape = input["shape"].tolist()
+ input_dtype = str(np.dtype(input["dtype"]))
+ shape_dict[input_name] = input_shape
+ dtype_dict[input_name] = input_dtype
+
+ #
+ # Save the model
+ #
+
+ #
+ # Load the TFLite Model for TVM:
+ #
+ # https://docs.tvm.ai/tutorials/frontend/from_tflite.html
+ # https://jackwish.net/tflite/docs/
+
+ model_buf = open(model_path, "rb").read()
+
+ #
+ # Get TFLite model from buffer
+ #
+ try:
+ import tflite
+
+ model = tflite.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model)
+ except AttributeError:
+ import tflite.Model
+
+ model = tflite.Model.Model.GetRootAsModel(model_buf, 0)
+ assert isinstance(model, tflite.Model.Model)
+
+ print("TVM: Importing a TFLite model ...")
+
+ return model, shape_dict, dtype_dict
+
+
+# ========================================================
+# extract_tflite_quantization
+# ========================================================
+
+
+def _make_qnn_params(quantization):
+ qnn_params = {}
+ qnn_params["min"] = quantization.MinAsNumpy()
+ qnn_params["max"] = quantization.MaxAsNumpy()
+ qnn_params["scale"] = quantization.ScaleAsNumpy()
+ qnn_params["zero_point"] = quantization.ZeroPointAsNumpy()
+ qnn_params["dim"] = quantization.QuantizedDimension()
+ # print(" Quantization: ({}, {}), s={}, z={}, dim={}".format(min, max, scale, zero_point, dim))
+ return qnn_params
+
+
+def extract_tflite_quantization(model):
+
+ assert model.SubgraphsLength() == 1, "only support one subgraph (main subgraph)"
+
+ subgraph = model.Subgraphs(0)
+
+ quantization_info = {}
+
+ # model inputs / outputs
+ model_inputs = subgraph.InputsAsNumpy()
+ model_outputs = subgraph.OutputsAsNumpy()
+
+ for node_id in model_inputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ dl_tensor_name = stm32.get_input_tensor_name(tensor_name)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ for node_id in model_outputs:
+ tensor = subgraph.Tensors(node_id)
+ tensor_name = tensor.Name().decode("utf-8")
+ tensor_type = tensor.Type()
+ #
+ # TODO: TVM does not preserve the output tensor names.
+ # Eventually, we should be able to form a valid name.
+ #
+ dl_tensor_name = stm32.get_output_tensor_name(tensor_name, 0)
+
+ quantization = tensor.Quantization()
+ if quantization is not None:
+ qnn_params = _make_qnn_params(quantization)
+ quantization_info[dl_tensor_name] = qnn_params
+
+ return quantization_info
+
+
+# ========================================================
+# run_tflite_model
+# ========================================================
+def run_tflite_model(model_path, image_data):
+ #
+ # Load TFLite model and allocate tensors.
+ #
+ interpreter = tf.lite.Interpreter(model_path=model_path)
+ interpreter.allocate_tensors()
+ #
+ # Get input and output tensors.
+ #
+ input_details = interpreter.get_input_details()[0]
+ output_details = interpreter.get_output_details()[0]
+
+ #
+ # Run test images
+ #
+ tf_results = np.empty(shape=[NUM_ITERATIONS, 10], dtype=np.float)
+ for i, image in enumerate(image_data):
+ #
+ # Normalize the input data
+ #
+ image = image / 255.0
+ image = scale_input_data(input_details, image)
+ interpreter.set_tensor(input_details["index"], image)
+ interpreter.invoke()
+ tf_results[i] = interpreter.get_tensor(output_details["index"])
+ tf_results[i] = scale_output_data(output_details, tf_results[i])
+
+ print(f"== [{i}] TFLite Output:")
+ print(tf_results[i])
+
+ return tf_results
+
+
+# ========================================================
+# run_tvm_model
+# ========================================================
+def run_tvm_model(build_dir, model_name, target_dir, image_path):
+
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+
+ tvm_results_name = os.path.join(build_dir, "tvm_results.txt")
+
+ #
+ # Build the model
+ #
+ tvm_dir = os.path.join(curr_path, "..", "..", "..")
+ test_dir = os.path.join(tvm_dir, "tests", "crt", "contrib", "stm32")
+
+ command = f"make -f {test_dir}/Makefile TVM_PATH={tvm_dir} MODEL_PATH={target_dir} BUILD_PATH={build_dir} IMAGE_PATH={image_path}"
+ print(f"{command}")
+ os.system(command)
+ #
+ # Run
+ #
+ command = f"{target_dir}/{model_name}.exe"
+ print(f"{command}")
+ os.system(command)
+
+ tvm_results = np.loadtxt(tvm_results_name)
+ print(f"== TVM Output:\n {tvm_results}")
+
+ #
+ # Clean temporary image files
+ #
+ if os.path.exists(tvm_results_name):
+ os.remove(tvm_results_name)
+
+ return tvm_results
+
+
+# ========================================================
+# check_network
+# ========================================================
+def check_network(build_dir, target_name, model_path, image_path):
+
+ model_name = "network"
+
+ model, shape_dict, dtype_dict = get_tflite_model(model_path)
+
+ #
+ # Generate random input data
+ #
+ image_data = []
+ for i in range(NUM_ITERATIONS):
+ assert len(shape_dict) == 1, "Only single input models are supported."
+ image_shape = list(shape_dict.values())[0]
+ in_data = np.random.randint(0, 255, size=image_shape).astype("uint8")
+ # Write raw data for using with the TVM implementation
+ filename = os.path.join(image_path, "{:02d}.raw".format(i))
+ dump_image(filename, in_data)
+ image_data.append(in_data)
+
+ mod, params = relay.frontend.from_tflite(model, shape_dict, dtype_dict)
+
+ #
+ # Build a TVM C module for the ARM CPU (without compiling the kernels
+ # library to the object code form):
+ #
+ with tvm.transform.PassContext(opt_level=3, config={"tir.disable_vectorize": True}):
+ rt_module = relay.build(mod, target="c -device=arm_cpu", params=params)
+
+ #
+ # Export model library format
+ #
+ target_dir = os.path.join(build_dir, target_name + "_gen")
+
+ if os.path.exists(target_dir):
+ print(f'Removing existing "{target_dir}" directory')
+ try:
+ shutil.rmtree(target_dir)
+ except OSError as err:
+ raise ValueError(f"emit_code.Error: {target_dir} : {err.strerror}")
+
+ mlf_tar_path = os.path.join(build_dir, target_name + "_lib.tar")
+ import tvm.micro as micro
+
+ micro.export_model_library_format(rt_module, mlf_tar_path)
+
+ emitter = stm32.CodeEmitter()
+ quantization = extract_tflite_quantization(model)
+ emitter.parse_library_format(mlf_tar_path, quantization)
+ emitter.emit_code(target_dir, model_name)
+
+ #
+ # Results
+ #
+ tf_results = run_tflite_model(model_path, image_data)
+ tvm_results = run_tvm_model(build_dir, model_name, target_dir, image_path)
+
+ check_result(tf_results, tvm_results)
+
+
+# ========================================================
+# check_result
+# ========================================================
+def check_result(tflite_results, tvm_results):
+ """Helper function to verify results"""
+
+ #
+ # MNIST quantized uint8 results in one single difference of
+ # ~ 0.004 so just escape this
+ #
+ ATOL = 1e-3
+ RTOL = 0.5
+
+ tvm.testing.assert_allclose(tflite_results, tvm_results, rtol=RTOL, atol=ATOL)
+
+
+# ========================================================
+# test_mnist
+# ========================================================
+def test_mnist():
+ curr_path = os.path.dirname(os.path.abspath(os.path.expanduser(__file__)))
+ model_path = os.path.join(curr_path, "models/mnist.tflite")
+ build_dir = os.path.join(curr_path, BUILD_DIR)
Review comment:
as we add more tests, we'll need to config filesystem modifications in a test-specific directory. can you do this:
```
DEBUG = False
tempdir_root = None
if DEBUG:
tempdir_root = os.path.join(curr_path, f"workspace", "test_mnist", datetime.datetime.now().strftime("%Y-%m-%dT%H-%M-%S"))
build_dir = tvm.contrib.utils.tempdir(tempdir_root)
```
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1557 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+
+ # TODO: '#include "stm32lib.h"\n\n' - the 'magic option is not contributed yet
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ for line in src.splitlines():
+ #
+ # This is sort of hacking - when AoT is available, we will be
+ # able to clean this ...
+ #
+ dst = dst + line + "\n"
+
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ #
+ # Constants:
+ #
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ #
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ #
+ self.activations_static = include_activations
+
+ #
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ #
+ if include_inputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs is True:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ #
+ # Parsed graph
+ #
+ self.nodes_ = []
+ self.arg_nodes_ = []
+ self.outputs_ = []
+ self.attrs_ = {}
+ self.node_row_ptr_ = []
+ #
+ # Parameters
+ #
+ self.params_ = {}
+ #
+ # Filled by data_placement()
+ #
+ self.weights_ = {}
+ self.activations_ = {}
+ self.input_data_ = {}
+ self.output_data_ = {}
+ self.nodes_size_ = 0
+ self.weights_size_ = 0
+ self.activations_size_ = 0
+
+ self.quantization_ = {}
+
+ def __extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self.input_data_:
+ if dl_tensor_name in quantization:
+ self.quantization_[dl_tensor_name] = quantization[dl_tensor_name]
+
+ #
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ #
+ assert len(self.output_data_) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self.output_data_:
+ for name in quantization:
+ if name not in self.input_data_:
+ self.quantization_["output"] = quantization[name]
+ break
+
+ def __get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self.nodes_[arg_nid]
+ arg_name = self.nodes_[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self.__get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def __tensor_is_output(self, nid, idx):
+ for out in self.outputs_:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def __get_tensor_from_node(self, nid, idx):
+ # 'eid' is index into the dltype', 'shape', etc.
+ eid = self.node_row_ptr_[nid] + idx
+ dltype = self.attrs_["dltype"][1][eid]
+ dims = self.attrs_["shape"][1][eid]
+ storage_id = self.attrs_["storage_id"][1][eid]
+ ndim = len(dims)
+ #
+ # Get tensor size
+ #
+ size = _get_tensor_size_bytes(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ #
+ # What is this byte_offset really ?
+ #
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ def __compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self.inputs_ = self.arg_nodes_.copy()
+
+ #
+ # weights:
+ #
+ offset = 0
+
+ for key in self.params_:
+ #
+ # First, find the node in graph
+ #
+ nid = 0
+ for node in self.nodes_:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self.__get_tensor_from_node(nid, 0)
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self.arg_nodes_:
+ node = self.nodes_[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self.inputs_.remove(idx)
+
+ self.weights_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ self.weights_size_ = offset
+
+ #
+ # activations:
+ #
+ buffer_list_ = {}
+
+ nid = 0
+ for node in self.nodes_:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self.__tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+ #
+ # Do not count the 'outputs_'
+ #
+ if self.__tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self.__get_tensor_from_node(nid, idx)
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self.activations_[dl_tensor_name] = tensor
+
+ self.nodes_size_ = self.nodes_size_ + 1
+
+ nid += 1
+
+ #
+ # Compute 'input_data_'
+ #
+ offset = 0
+ for nid in self.inputs_:
+ node = self.nodes_[nid]
+ node_name = node["name"]
+ #
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ #
+ dl_tensor_name = get_input_tensor_name(node_name)
+ #
+ # This tensor is at some index inside 'input_data_' dictionary
+ # depending on the 'inputs_' list order. We refer to this position
+ # when generating the XXX.h file.
+ #
+ tensor = self.__get_tensor_from_node(nid, 0)
+
+ if self.inputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.input_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Compute 'output_data_'
+ #
+ offset = 0
+ for output in self.outputs_:
+ nid = output[0]
+ idx = output[1]
+
+ node = self.nodes_[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self.__get_tensor_from_node(nid, idx)
+
+ if self.outputs_static == True:
+ #
+ # Remember this tensor with the storage id
+ #
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+ #
+ # Compute the offset
+ #
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self.output_data_[dl_tensor_name] = tensor
+
+ #
+ # Next offset
+ #
+ offset = aligned_offset + tensor["size"]
+
+ #
+ # Go over all storage IDs and compute offsets and activations_size_
+ #
+ offset = 0
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self.activations_size_ = offset
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
Review comment:
ok, i don't think we need to block on this but i don't see you doing any modification to `self._nodes`, `self._arg_nodes`, `self._node_row_ptr`, `self._outputs`, or `self._attrs` right now.
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1369 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ dst = dst + src
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ self.activations_static = include_activations
+
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ if include_inputs:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ # Parsed graph
+ self._nodes = []
+ self._arg_nodes = []
+ self._outputs = []
+ self._attrs = {}
+ self._node_row_ptr = []
+
+ # Parameters
+ self._params = {}
+
+ # Filled by data_placement()
+ self._weights = {}
+ self._activations = {}
+ self._input_data = {}
+ self._output_data = {}
+ self._nodes_size = 0
+ self._weights_size = 0
+ self._activations_size = 0
+
+ self._quantization = {}
+
+ def _extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self._input_data:
+ if dl_tensor_name in quantization:
+ self._quantization[dl_tensor_name] = quantization[dl_tensor_name]
+
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ assert len(self._output_data) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self._output_data:
+ for name in quantization:
+ if name not in self._input_data:
+ self._quantization["output"] = quantization[name]
+ break
+
+ def _get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self._nodes[arg_nid]
+ arg_name = self._nodes[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self._get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def _tensor_is_output(self, nid, idx):
+ for out in self._outputs:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def _get_tensor_from_node(self, nid, idx):
+ # 'eid' is index into the dltype', 'shape', etc.
+ eid = self._node_row_ptr[nid] + idx
+ dltype = self._attrs["dltype"][1][eid]
+ dims = self._attrs["shape"][1][eid]
+ storage_id = self._attrs["storage_id"][1][eid]
+ ndim = len(dims)
+ size = _get_tensor_size_bytes(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ def _compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self._inputs = self._arg_nodes.copy()
+
+ # weights:
+ offset = 0
+
+ for key in self._params:
+
+ # First, find the node in graph
+ nid = 0
+ for node in self._nodes:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self._get_tensor_from_node(nid, 0)
+
+ # Compute the offset
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self._arg_nodes:
+ node = self._nodes[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self._inputs.remove(idx)
+
+ self._weights[dl_tensor_name] = tensor
+
+ # Next offset
+ offset = aligned_offset + tensor["size"]
+
+ self._weights_size = offset
+
+ # activations:
+ buffer_list_ = {}
+
+ nid = 0
+ for node in self._nodes:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self._tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+
+ # Do not count the '_outputs'
+ if self._tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self._get_tensor_from_node(nid, idx)
+
+ # Remember this tensor with the storage id
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self._activations[dl_tensor_name] = tensor
+
+ self._nodes_size = self._nodes_size + 1
+
+ nid += 1
+
+ # Compute '_input_data'
+ offset = 0
+ for nid in self._inputs:
+ node = self._nodes[nid]
+ node_name = node["name"]
+
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ dl_tensor_name = get_input_tensor_name(node_name)
+
+ # This tensor is at some index inside '_input_data' dictionary
+ # depending on the '_inputs' list order. We refer to this position
+ # when generating the XXX.h file.
+ tensor = self._get_tensor_from_node(nid, 0)
+
+ if self.inputs_static:
+
+ # Remember this tensor with the storage id
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+
+ # Compute the offset
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self._input_data[dl_tensor_name] = tensor
+
+ # Next offset
+ offset = aligned_offset + tensor["size"]
+
+ # Compute '_output_data'
+ offset = 0
+ for output in self._outputs:
+ nid = output[0]
+ idx = output[1]
+
+ node = self._nodes[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self._get_tensor_from_node(nid, idx)
+
+ if self.outputs_static:
+
+ # Remember this tensor with the storage id
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+
+ # Compute the offset
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self._output_data[dl_tensor_name] = tensor
+
+ # Next offset
+ offset = aligned_offset + tensor["size"]
+
+ # Go over all storage IDs and compute offsets and _activations_size
+ offset = 0
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self._activations_size = offset
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self._nodes = self.graph_["nodes"]
+ elif key == "arg_nodes":
+ self._arg_nodes = self.graph_["arg_nodes"]
+ elif key == "node_row_ptr":
+ self._node_row_ptr = self.graph_["node_row_ptr"]
+ elif key == "heads":
+ self._outputs = self.graph_["heads"]
+ elif key == "attrs":
+ self._attrs = self.graph_["attrs"]
+ elif key == "metadata":
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ # Build all tensor lists
+ self._compute_data_placement()
+
+ # Extract quantization info for inputs/outputs
+ if quantization is not None:
+ self._extract_quantization_info(quantization)
+
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The ModuleLibraryFormat object to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ # Extract informations from the Model Library Format
+ graph_file = os.path.join(extract_path, "executor-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+
+ # Map -> Python Dict
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ # Sort params for debugging
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+
+ self.graph_ = graph_dict
+ self._params = params_dict
+ self.lib_ = src_files
+
+ self._parse_model(quantization)
+
+ def parse_module(self, module, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM Runtime Module
+ The module to parse.
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ graph = module.get_json()
+ if not isinstance(graph, (str,)):
+ try:
+ graph = graph._tvm_graph_json()
+ except AttributeError:
+ raise ValueError("Type %s is not supported" % type(graph))
+
+ # Sort params for debugging
+ params_dict = {}
+ tmp_params = module.get_params()
+ for k in sorted(tmp_params.keys()):
+ params_dict[k] = tmp_params[k]
+
+ self.graph_ = json.loads(graph)
+ self._params = params_dict
+ self.lib_ = module.get_lib()
+
+ self._parse_model(quantization)
+
+ def _emit_params_data(self, name, out_h, out_c):
+ """ Emits the network_data[c,h] files with parameters."""
+
+ name_upper = name.upper()
+
+ # XXX_data.h
+
+ out_h.write(
+ textwrap.dedent(
+ f"""\
+ #ifndef __{name_upper}_DATA_H_
+ #define __{name_upper}_DATA_H_
+
+ #include \"ai_runtime_api.h\"
+
+ AI_API_ENTRY
+ const ai_ptr ai_{name}_data_weights_get (void);
+
+ #endif /* __{name_upper}_DATA_H_ */
+ """
+ )
+ )
+
+ # XXX_data.cc
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ #include \"{name}_data.h\"
+
+ const ai_ptr ai_{name}_data_weights_get (void)
+ {{
+ AI_ALIGNED({self.DATA_ALIGNMENT_BYTES}) static const __attribute__ ((section(\".nn_weights\"))) uint8_t s_{name}_weights[] = {{
+ """
+ )
+ )
+
+ # Weights are arranged in the order of 'params_'
+ offset = 0
+
+ for key in self._params:
+ data = self._params[key] # ND Array
+ npdata = data.asnumpy()
+ blob = npdata.tobytes()
+
+ out_c.write(f'// "{key}": \n')
+ out_c.write(f"\t")
+
+ count = 0
+
+ # Align by emitting garbage between un-aligned data
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self._weights[dl_tensor_name]
+ tensor_offset = tensor["offset"]
+ tensor_size = tensor["size"]
+
+ while offset < tensor_offset:
+ count += 1
+ out_c.write("0x{:02X}, ".format(0))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+ offset += 1
+
+ for val in blob:
+ count += 1
+ out_c.write("0x{:02X}, ".format(val))
+ if count == 12:
+ out_c.write("\n\t")
+ count = 0
+
+ offset += tensor_size
+
+ out_c.write(f"\n")
+
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ }};
+ return (const ai_ptr)s_{name}_weights;
+ }}
+ """
+ )
+ )
+
+ def _emit_open(self, name, out_h, out_c):
+ """Emits the network.h file with a few network defines and
+ writes the header part of the network.c file."""
+
+ name_upper = name.upper()
+
+ input_size = len(self._input_data)
+ output_size = len(self._output_data)
+
+ # XXX.h
+
+ out_h.write(
+ textwrap.dedent(
+ f"""\
+ #ifndef __AI_{name_upper}_H__
+ #define __AI_{name_upper}_H__
+
+ #include \"ai_runtime_api.h\"
+
+ #define _{name_upper}_INPUTS_COUNT_ ({input_size})
+ #define _{name_upper}_OUTPUTS_COUNT_ ({output_size})
+ #define _{name_upper}_ACTIVATION_BYTES_ ({self._activations_size})
+ """
+ )
+ )
+
+ # XXX.c
+
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ #include <stdio.h>
+
+ #include \"dlpack/dlpack.h\"
+ #include \"tvm/runtime/c_runtime_api.h\"
+ #include \"{name}.h\"
+ #include \"{name}_data.h\"
+ """
+ )
+ )
+
+ def _emit_close(self, name, out_h, out_c):
+ """ Emits the ai_model_info structure. """
+
+ name_upper = name.upper()
+
+ # datetime object containing current date and time
+ now = datetime.now()
+ # dd/mm/YY H:M:S
+ dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
+
+ # XXX.h
+
+ out_h.write(f"#endif /*__AI_{name_upper}_H__*/ \n")
+
+ # XXX.c
+
+ if self.activations_static:
+ out_c.write(
+ f'AI_ALIGNED({self.DATA_ALIGNMENT_BYTES}) __attribute__ ((section(".{name}.nn_data_act"))) uint8_t {name}_activations[{self._activations_size}];\n'
+ )
+ else:
+ out_c.write(f"AI_STATIC ai_ptr {name}_activations = NULL;")
+
+ # Emit network structure
+ num_inputs = len(self._input_data)
+ num_outputs = len(self._output_data)
+
+ tool_version = tvm.__version__
+ api_version = f"{AI_API_VERSION_MAJOR}.{AI_API_VERSION_MINOR}.{AI_API_VERSION_MICRO}.0"
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ AI_API_ENTRY __attribute__ ((section(".nn_models"))) ai_model_info {name}_network = {{
+ .name = \"{name}\",
+ .datetime = \"{dt_string}\",
+ .revision = \"{AI_TOOLS_REVISION}\",
+ .tool_version = \"{tool_version}\",
+ .api_version = \"{api_version}\",
+ .n_nodes = {self._nodes_size},
+ .n_inputs = {num_inputs},
+ .n_outputs = {num_outputs},
+ .activations_size = {self._activations_size},
+ .params_size = {self._weights_size},
+ .activations = {name}_activations,
+ .inputs = _InputsList,
+ .outputs = _OutputsList,
+ .ai_get_params = &ai_{name}_data_weights_get,
+ .ai_create = &ai_{name}_create,
+ .ai_destroy = &ai_{name}_destroy,
+ .ai_run = &ai_{name}_run
+ }};
+ """
+ )
+ )
+
+ def _emit_tensor_shape(self, dl_tensor_name, ndim, shape, strides, out_c):
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_shape[{ndim}] = {{{shape[1:-1]}}}; \n")
+ assert strides is None, f"###Error: non-compact tensors are not handled yet."
+ out_c.write(f"AI_STATIC int64_t {dl_tensor_name}_strides[{ndim}] = {{}}; \n")
+
+ def _emit_tensor_quant(self, dl_tensor_name, out_c):
+
+ if dl_tensor_name in self._quantization:
+ quantization = self._quantization[dl_tensor_name]
+
+ # At this time, TVM only supports quantization info with
+ # single output models.
+ elif dl_tensor_name in self._output_data and "output" in self._quantization.keys():
+ quantization = self._quantization["output"]
+ else:
+ quantization = None
+
+ if quantization is not None:
+ scale = quantization["scale"]
+ zero_point = quantization["zero_point"]
+
+ # Sometimes we get a scalar with ScaleAsNumpy.
+ # This seem to mean not quantized ?
+ if not isinstance(scale, np.ndarray):
+ assert scale == 0.0, f"Non-quantized tensor with scale != 0.0"
+ assert (
+ not isinstance(zero_point, np.ndarray) and zero_point == 0
+ ), f"Non-quantized tensor with zero_point != 0"
+ return None
+
+ scale_size = len(scale)
+ zero_point_size = len(zero_point)
+
+ assert len(scale) == len(
+ zero_point
+ ), f"Inconsistent quantizations scale:{scale} vs zero-point:{zero_point}"
+
+ if len(scale) == 1:
+ quant_name = dl_tensor_name + "_quant"
+
+ out_c.write(f"AI_STATIC float {quant_name}_scale[{scale_size}] = {{ ")
+ for val in scale:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};\n")
+ out_c.write(f"AI_STATIC int32_t {quant_name}_zero_point[{zero_point_size}] = {{ ")
+ for val in zero_point:
+ out_c.write(f"{val}, ")
+ out_c.write(f"}};")
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ AI_STATIC ai_quantization_info {quant_name} = {{
+ .scale = {quant_name}_scale,
+ .zero_point = {quant_name}_zero_point,
+ .dim = -1
+ }};
+ """
+ )
+ )
+
+ return quant_name
+
+ return None
+
+ def _emit_tensor_init(self, dl_tensor_name, tensor, out_c):
+ """ Emits the tensor instantiation code. """
+
+ dltype = tensor["dltype"]
+ dims = tensor["dims"]
+ strides = tensor["strides"]
+ byte_offset = tensor["byte_offset"]
+ dtype = _get_type_data(dltype)
+ ndim = len(dims)
+ shape = str(dims)
+ self._emit_tensor_shape(dl_tensor_name, ndim, shape, strides, out_c)
+
+ # Quantization
+ quant_name = self._emit_tensor_quant(dl_tensor_name, out_c)
+
+ # Contents
+ #
+ # TODO: use the 'storage_id':
+ # " .ctx = {{ {} }}, \n".format(str(storage_id)[1:-1])
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ AI_ALIGNED({self.DATA_ALIGNMENT_BYTES}) AI_STATIC ai_tensor {dl_tensor_name} = {{
+ .dltensor = {{
+ .data = (ai_ptr)(NULL),
+ .device = {{kDLCPU,0}},
+ .ndim = {ndim},
+ .dtype = {{{dtype}}},
+ .shape = {dl_tensor_name}_shape,
+ .strides = {dl_tensor_name}_strides,
+ .byte_offset = {byte_offset}
+ }},
+ """
+ )
+ )
+
+ # Figure out quantization, if exists
+ if quant_name is not None:
+ out_c.write(f" .quant = &{quant_name} \n")
+ else:
+ out_c.write(f" .quant = NULL \n")
+ out_c.write(f"}}; \n")
+
+ def _emit_activation_buffers(self, name, out_c):
+ # pylint: disable=unused-argument
+ """ Emits activation tensors, including inputs/outputs."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ //
+ // Inputs:
+ //
+ """
+ )
+ )
+
+ # shape/buffer
+ for dl_tensor_name in self._input_data:
+ tensor = self._input_data[dl_tensor_name]
+ self._emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ # tensor
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _InputsList[] = {{ \n")
+ for dl_tensor_name in self._input_data:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ //
+ // Activations:
+ //
+ """
+ )
+ )
+ for dl_tensor_name in self._activations:
+ tensor = self._activations[dl_tensor_name]
+ self._emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ # Outputs:
+ out_c.write(
+ textwrap.dedent(
+ f"""\
+ //
+ // Outputs:
+ //
+ """
+ )
+ )
+ for dl_tensor_name in self._output_data:
+ tensor = self._output_data[dl_tensor_name]
+ self._emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+ out_c.write(f"\n")
+
+ idx = 0
+ out_c.write(f"AI_STATIC ai_tensor * _OutputsList[] = {{ \n")
+ for dl_tensor_name in self._output_data:
+ out_c.write(f" &{dl_tensor_name}, // [{idx}]\n")
+ idx = idx + 1
+ out_c.write(f"}}; \n")
+ out_c.write(f"\n")
+
+ def _emit_params_buffers(self, name, out_c):
+ """ Emits all parameter tensors."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ //
+ // Weights: {name}
+ //
+ """
+ )
+ )
+ for dl_tensor_name in self._weights:
+ tensor = self._weights[dl_tensor_name]
+ self._emit_tensor_init(dl_tensor_name, tensor, out_c)
+ out_c.write(f"\n")
+
+ def _emit_network(self, name, out_c):
+ """ Emits prototypes for the network operator functions."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ //
+ // Network: {name}
+ //
+ """
+ )
+ )
+ for node in self._nodes:
+ if node["op"] == "null":
+ continue
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ if func_name == "__nop":
+ continue
+
+ out_c.write(
+ f"TVM_DLL int32_t {func_name}(void * args, void * arg_type_ids, int32_t num_args); \n"
+ )
+ out_c.write(f"\n")
+
+ def _emit_tensor_activation(self, dl_tensor_name, tensor, out_c):
+
+ storage_id = tensor["storage_id"]
+ offset = tensor["offset"]
+ out_c.write(
+ textwrap.indent(
+ textwrap.dedent(
+ f"""
+ //
+ // {dl_tensor_name}: storage_id:{storage_id}
+ //
+ {dl_tensor_name}.dltensor.data = (ai_ptr)(activations + {offset});
+ """
+ ),
+ " ",
+ )
+ )
+
+ def _emit_activation_init(self, name, out_c):
+ """ Emits buffer initialization code for activation tensors."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // {name}_configure_activations
+ // {DBAR}
+ AI_STATIC AI_INLINE
+ ai_status {name}_configure_activations (
+ const ai_ptr activations
+ )
+ {{
+ if (activations == NULL) {{
+ TVMAPISetLastError (\"Non-null activations arena is required for this model.\");
+ return AI_STATUS_ERROR;
+ }}
+ """
+ )
+ )
+
+ # Allocate inputs with the static model
+ if self.inputs_static:
+ for dl_tensor_name in self._input_data:
+ tensor = self._input_data[dl_tensor_name]
+ self._emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ # Prepare activation buffers
+ for dl_tensor_name in self._activations:
+ tensor = self._activations[dl_tensor_name]
+ self._emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ # Allocate outputs with the static model
+ if self.outputs_static:
+ for dl_tensor_name in self._output_data:
+ tensor = self._output_data[dl_tensor_name]
+ self._emit_tensor_activation(dl_tensor_name, tensor, out_c)
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
+
+ def _emit_params_init(self, name, out_c):
+ """ Emits buffer initialization code for params tensors."""
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // {name}_configure_weights
+ // {DBAR}
+ AI_STATIC AI_INLINE
+ ai_status {name}_configure_weights (
+ const ai_ptr weights
+ )
+ {{
+ if (weights == NULL) {{
+ TVMAPISetLastError(\"Non-null weights arena is required for this model.\");
+ return AI_STATUS_ERROR;
+ }}
+ """
+ )
+ )
+
+ for dl_tensor_name in self._weights:
+ tensor = self._weights[dl_tensor_name]
+ offset = tensor["offset"]
+ out_c.write(
+ textwrap.indent(
+ textwrap.dedent(
+ f"""\
+ //
+ // {dl_tensor_name}
+ //
+ {dl_tensor_name}.dltensor.data = (ai_ptr)(weights + {offset});
+ """
+ ),
+ " ",
+ )
+ )
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
+
+ def _emit_init(self, name, out_c):
+ """ Emits buffer initialization code."""
+
+ self._emit_activation_init(name, out_c)
+ self._emit_params_init(name, out_c)
+
+ def _emit_run(self, name, out_h, out_c):
+ """ Emits the run function code."""
+
+ out_h.write(
+ textwrap.dedent(
+ f"""
+ AI_API_ENTRY
+ ai_status ai_{name}_run (
+ ai_tensor *inputs[],
+ ai_tensor *outputs[]
+ );
+ """
+ )
+ )
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // ai_{name}_run
+ // {DBAR}
+ AI_API_ENTRY
+ ai_status ai_{name}_run (
+ ai_tensor *inputs[],
+ ai_tensor *outputs[]
+ )
+ {{
+ """
+ )
+ )
+
+ # Execute nodes one by one
+ nid = 0
+
+ for node in self._nodes:
+ node_name = node["name"]
+ node_name_upper = node_name.upper()
+
+ nid += 1
+
+ if node["op"] == "null":
+ continue
+
+ assert node["op"] == "tvm_op", f"###Error: Only TVM ops are supported."
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+
+ if func_name == "__nop":
+ continue
+
+ out_c.write(f" // \n")
+ out_c.write(f" // {func_name}\n")
+ out_c.write(f" // \n")
+
+ # Prepare TVM packed function - this is the one called
+ if name == "__nop":
+ print(" exec: __nop")
+ continue
+
+ if name == "__copy":
+ print(" exec: __copy")
+ continue
+
+ # Get function from the TVM module
+ #
+ # void * args : arg_values.data()
+ # void * arg_type_ids : arg_tcodes.data()
+ # int32_t num_args : arg_values.size()
+
+ dl_args_name = _get_node_args_name(node_name)
+ dl_arg_types_name = _get_node_arg_types_name(node_name)
+
+ num_inputs = len(node["inputs"])
+ num_outputs = int(node_attrs["num_outputs"])
+ num_args = num_inputs + num_outputs
+
+ out_c.write(f" TVMValue {dl_args_name}[{num_args}]; \n")
+ out_c.write(f" int32_t {dl_arg_types_name}[{num_args}]; \n")
+
+ curr_idx = 0
+
+ for arg in node["inputs"]:
+ dl_tensor_name = self._get_node_arg_name(arg)
+ #
+ # If this input is not an activation or a parameter => find the input
+ #
+ if dl_tensor_name not in self._weights and dl_tensor_name not in self._activations:
+
+ assert dl_tensor_name in self._input_data, "Tensor {} not registered ?".format(
+ dl_tensor_name
+ )
+
+ input_idx = 0
+ for dl_entry_name in self._input_data:
+ if dl_entry_name == dl_tensor_name:
+ break
+ input_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &inputs[{input_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+
+ curr_idx += 1
+
+ for idx in range(num_outputs):
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ # If this output is not an activation => find the output
+ if dl_tensor_name not in self._activations:
+
+ assert dl_tensor_name in self._output_data
+
+ output_idx = 0
+ for dl_exit_name in self._output_data:
+ if dl_exit_name == dl_tensor_name:
+ break
+ output_idx += 1
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &outputs[{output_idx}]->dltensor; \n"
+ )
+ else:
+ out_c.write(
+ f" {dl_args_name}[{curr_idx}].v_handle = &{dl_tensor_name}.dltensor; \n"
+ )
+ out_c.write(f" {dl_arg_types_name}[{curr_idx}] = kTVMNDArrayHandle; \n")
+ out_c.write(f"\n")
+
+ curr_idx += 1
+
+ # call this function
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ #if (_VERBOSE_ > 0)
+ printf (\" {func_name} ... \\r\\n\");
+ #endif
+ if ({func_name} ({dl_args_name}, {dl_arg_types_name}, {num_args})) {{
+ TVMAPISetLastError("Invalid handle");
+ return AI_STATUS_ERROR;
+ }}
+ #if (_VERBOSE_ > 0)
+ printf (\" {func_name} Done.\\r\\n\");
+ #endif
+ """
+ )
+ )
+ out_c.write(f"\n")
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
+ out_c.write(f"\n")
+
+ def _emit_create_destroy(self, name, out_h, out_c):
+ """ Emits the create/destroy functions."""
+
+ out_h.write(
+ textwrap.dedent(
+ f"""
+ AI_API_ENTRY
+ ai_status ai_{name}_create (
+ const ai_ptr weights,
+ const ai_ptr activations
+ );
+ """
+ )
+ )
+
+ out_h.write(
+ textwrap.dedent(
+ f"""
+ AI_API_ENTRY
+ ai_status ai_{name}_destroy ();
+ """
+ )
+ )
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // ai_{name}_create
+ // {DBAR}
+ AI_API_ENTRY
+ ai_status ai_{name}_create(
+ const ai_ptr weights,
+ const ai_ptr activations
+ )
+ {{
+ ai_status status = AI_STATUS_OK;
+ status = {name}_configure_weights (weights);
+ if (status != AI_STATUS_OK) {{
+ return status;
+ }}
+ status = {name}_configure_activations (activations);
+ if (status != AI_STATUS_OK) {{
+ return status;
+ }}
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
+
+ out_c.write(
+ textwrap.dedent(
+ f"""
+ // {DBAR}
+ // ai_{name}_destroy
+ // {DBAR}
+ AI_API_ENTRY
+ ai_status ai_{name}_destroy ()
+ {{
+ return AI_STATUS_OK;
+ }}
+ """
+ )
+ )
+
+ def emit_code(self, dest_dir, model_name):
+ """ Emits the C code implementing the model. """
+
+ # Build the directory structure
+ if os.path.exists(dest_dir):
+ raise ValueError(f"emit_code.Error: {dest_dir} exists.")
+
+ # Make a new one
+ os.makedirs(dest_dir)
+
+ # Fix the model name
+ model_name = re.sub("[^0-9a-zA-Z_]+", "_", model_name)
+ model_name = model_name.lower()
+
+ # Write the C code: we can parse the string
+ if isinstance(self.lib_, list):
+ # List of strings from Model Library Format C files
+ for idx, src in enumerate(self.lib_):
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib{idx}.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+ else:
+ # a TVM RuntimeGraphFactory
+ src = self.lib_.get_source(fmt="c")
+ code = _preprocess_code(src)
+ filename = os.path.join(dest_dir, f"{model_name}_lib.c")
+ with open(filename, "w") as fout:
+ fout.write(code)
+
+ # Save params as binary data
+ saved_params = tvm.runtime.save_param_dict(self._params)
+ params_name = os.path.join(dest_dir, model_name + ".params")
+ with open(params_name, "wb") as f:
+ f.write(saved_params)
+
+ # Write the .json
+ graph_name = os.path.join(dest_dir, model_name + ".json")
+ json_string = json.dumps(self.graph_, indent=4)
+ with open(graph_name, "w") as f:
+ print(json_string, file=f)
+
+ # emit X_data[c,h]
+ data_h_name = os.path.join(dest_dir, model_name + "_data.h")
+ data_c_name = os.path.join(dest_dir, model_name + "_data.c")
+ model_h_name = os.path.join(dest_dir, model_name + ".h")
+ model_c_name = os.path.join(dest_dir, model_name + ".c")
+
+ data_h = open(data_h_name, "w")
Review comment:
could you do this like so:
```
import contextlib
with contextlib.ExitStack() as exit_stack:
data_h = exit_stack.enter_context(open(data_h_name, "w"))
data_c = exit_stack.enter_context(open(data_c_name, "w"))
out_h = exit_stack.enter_context(open(out_h_name, "w"))
out_c = exit_stack.enter_context(open(out_h_name, "w"))
self._emit_params_data(...)
self.emit_open(...)
# .. all other self._emit
# no need to close() the ExitStack does this for you
```
this way, if an exception occurs in any `self._emit_*`, Python will auto-close the files for you.
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1369 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
+ """ Hack the C code implementing the model. """
+ dst = "#include <stdio.h>\n" "#include <math.h>\n\n"
+ dst = dst + src
+ return dst
+
+
+class CodeEmitter(object):
+ """Code emitter class."""
+
+ DATA_ALIGNMENT_BYTES = 8
+
+ def __init__(self, include_activations=True, include_inputs=True, include_outputs=True):
+ """Initialize the Emitter instance.
+
+ Parameters
+ ----------
+ include_activations:
+ The Emitter allocates the storage for the activations data
+ and places it in a specific data section. If Falsr, the
+ main application is responsible for allocating the activations
+ storage. Default: True.
+
+ include_inputs/include_outputs:
+ The Emitter allocates the storage for the input/output data.
+ This storage is shared with the activations and placed in the
+ specific activations data section. If False, the main
+ application is responsible for allocating the input/output
+ data storage. Default: True.
+
+ Returns
+ -------
+ CodeEmitter object.
+
+ """
+
+ # Static model: activations placed into a nn_data_act section
+ # Dynamic model: activations need to be malloc'ed by the
+ # applications.
+ self.activations_static = include_activations
+
+ # Inputs/outputs may be allocated within the activations or
+ # separately.
+ # TODO: Separate the inputs from activations inside TVM.
+ if include_inputs:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static inputs are not allowed without activations."
+ self.inputs_static = include_inputs
+
+ if include_outputs:
+ assert (
+ self.activations_static == True
+ ), "###Error: Static outputs are not allowed without activations."
+ self.outputs_static = include_outputs
+
+ # Parsed graph
+ self._nodes = []
+ self._arg_nodes = []
+ self._outputs = []
+ self._attrs = {}
+ self._node_row_ptr = []
+
+ # Parameters
+ self._params = {}
+
+ # Filled by data_placement()
+ self._weights = {}
+ self._activations = {}
+ self._input_data = {}
+ self._output_data = {}
+ self._nodes_size = 0
+ self._weights_size = 0
+ self._activations_size = 0
+
+ self._quantization = {}
+
+ def _extract_quantization_info(self, quantization):
+ """ Build dictionary with quantization infos."""
+
+ for dl_tensor_name in self._input_data:
+ if dl_tensor_name in quantization:
+ self._quantization[dl_tensor_name] = quantization[dl_tensor_name]
+
+ # Matching outputs is more difficult because TVM does not preserve
+ # output tensor names.
+ # We only support models with a single output now.
+ assert len(self._output_data) == 1, "Multiple outputs models are not yet supported."
+
+ for dl_tensor_name in self._output_data:
+ for name in quantization:
+ if name not in self._input_data:
+ self._quantization["output"] = quantization[name]
+ break
+
+ def _get_node_arg_name(self, arg):
+ arg_nid = arg[0]
+ arg_idx = arg[1]
+ arg_node = self._nodes[arg_nid]
+ arg_name = self._nodes[arg_nid]["name"]
+ if arg_node["op"] == "null":
+ # parameter
+ dl_tensor_name = get_input_tensor_name(arg_name)
+ elif arg_node["name"] == "reshape_nop":
+ # Handle __nop
+ src = arg_node["inputs"][0]
+ dl_tensor_name = self._get_node_arg_name(src)
+ else:
+ # activation
+ dl_tensor_name = get_output_tensor_name(arg_name, arg_idx)
+ return dl_tensor_name
+
+ def _tensor_is_output(self, nid, idx):
+ for out in self._outputs:
+ out_nid = out[0]
+ out_idx = out[1]
+ if out_nid == nid and out_idx == idx:
+ return True
+ return False
+
+ def _get_tensor_from_node(self, nid, idx):
+ # 'eid' is index into the dltype', 'shape', etc.
+ eid = self._node_row_ptr[nid] + idx
+ dltype = self._attrs["dltype"][1][eid]
+ dims = self._attrs["shape"][1][eid]
+ storage_id = self._attrs["storage_id"][1][eid]
+ ndim = len(dims)
+ size = _get_tensor_size_bytes(dims, dltype)
+
+ tensor = {
+ "dltype": dltype,
+ "ndim": ndim,
+ "dims": dims,
+ "strides": None,
+ "storage_id": storage_id,
+ "byte_offset": 0,
+ "offset": 0,
+ "size": size,
+ }
+
+ return tensor
+
+ def _compute_data_placement(self):
+ """ Compute inputs, outputs, weight, activation sizes"""
+
+ self._inputs = self._arg_nodes.copy()
+
+ # weights:
+ offset = 0
+
+ for key in self._params:
+
+ # First, find the node in graph
+ nid = 0
+ for node in self._nodes:
+ if node["name"] == key:
+ break
+ nid += 1
+
+ dl_tensor_name = get_input_tensor_name(key)
+ tensor = self._get_tensor_from_node(nid, 0)
+
+ # Compute the offset
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ for idx in self._arg_nodes:
+ node = self._nodes[idx]
+ node_name = node["name"]
+ if node_name == key:
+ self._inputs.remove(idx)
+
+ self._weights[dl_tensor_name] = tensor
+
+ # Next offset
+ offset = aligned_offset + tensor["size"]
+
+ self._weights_size = offset
+
+ # activations:
+ buffer_list_ = {}
+
+ nid = 0
+ for node in self._nodes:
+
+ if node["op"] == "null":
+ nid += 1
+ continue
+
+ if node["op"] != "tvm_op":
+ raise ValueError(f"Only TVM ops are supported")
+
+ node_name = node["name"]
+ node_attrs = node["attrs"]
+ func_name = node_attrs["func_name"]
+ num_outputs = int(node_attrs["num_outputs"])
+
+ if func_name == "__nop":
+ assert node_name == "reshape_nop", f"Unsupported __nop operator {node_name}."
+ assert num_outputs == 1
+ assert not self._tensor_is_output(nid, 0)
+ nid += 1
+ continue
+
+ for idx in range(num_outputs):
+
+ # Do not count the '_outputs'
+ if self._tensor_is_output(nid, idx):
+ continue
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+ tensor = self._get_tensor_from_node(nid, idx)
+
+ # Remember this tensor with the storage id
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+
+ self._activations[dl_tensor_name] = tensor
+
+ self._nodes_size = self._nodes_size + 1
+
+ nid += 1
+
+ # Compute '_input_data'
+ offset = 0
+ for nid in self._inputs:
+ node = self._nodes[nid]
+ node_name = node["name"]
+
+ # Arthur: I suppose that input nodes only have a single
+ # output dependency
+ dl_tensor_name = get_input_tensor_name(node_name)
+
+ # This tensor is at some index inside '_input_data' dictionary
+ # depending on the '_inputs' list order. We refer to this position
+ # when generating the XXX.h file.
+ tensor = self._get_tensor_from_node(nid, 0)
+
+ if self.inputs_static:
+
+ # Remember this tensor with the storage id
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+
+ # Compute the offset
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self._input_data[dl_tensor_name] = tensor
+
+ # Next offset
+ offset = aligned_offset + tensor["size"]
+
+ # Compute '_output_data'
+ offset = 0
+ for output in self._outputs:
+ nid = output[0]
+ idx = output[1]
+
+ node = self._nodes[nid]
+ node_name = node["name"]
+
+ dl_tensor_name = get_output_tensor_name(node_name, idx)
+
+ tensor = self._get_tensor_from_node(nid, idx)
+
+ if self.outputs_static:
+
+ # Remember this tensor with the storage id
+ storage_id = tensor["storage_id"]
+ if storage_id not in buffer_list_:
+ buffer_list_[storage_id] = []
+ buffer_entry = buffer_list_[storage_id]
+ buffer_entry.append(tensor)
+ else:
+
+ # Compute the offset
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+
+ self._output_data[dl_tensor_name] = tensor
+
+ # Next offset
+ offset = aligned_offset + tensor["size"]
+
+ # Go over all storage IDs and compute offsets and _activations_size
+ offset = 0
+ for storage_id in buffer_list_:
+ buffer_entry = buffer_list_[storage_id]
+
+ new_offset = offset
+ for tensor in buffer_entry:
+ assert tensor["storage_id"] == storage_id
+ dltype = tensor["dltype"]
+ aligned_offset = _get_aligned_offset(offset, dltype)
+ tensor["offset"] = aligned_offset
+ size = tensor["size"]
+ if (aligned_offset + size) > new_offset:
+ new_offset = aligned_offset + size
+ offset = new_offset
+
+ self._activations_size = offset
+
+ def _parse_model(self, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ module : TVM module or ModuleLibraryFormat object
+ The module to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ for key in self.graph_:
+ if key == "nodes":
+ self._nodes = self.graph_["nodes"]
+ elif key == "arg_nodes":
+ self._arg_nodes = self.graph_["arg_nodes"]
+ elif key == "node_row_ptr":
+ self._node_row_ptr = self.graph_["node_row_ptr"]
+ elif key == "heads":
+ self._outputs = self.graph_["heads"]
+ elif key == "attrs":
+ self._attrs = self.graph_["attrs"]
+ elif key == "metadata":
+ continue
+ else:
+ print("### Error: JSON key {} not supported".format(key))
+ assert False
+
+ # Build all tensor lists
+ self._compute_data_placement()
+
+ # Extract quantization info for inputs/outputs
+ if quantization is not None:
+ self._extract_quantization_info(quantization)
+
+ def parse_library_format(self, model_library_format_path, quantization=None):
+ """Parse the module. Build internal data structures.
+
+ Parameters
+ ----------
+ model_library_format_path :
+ The ModuleLibraryFormat object to parse
+
+ quantization: Dictionary
+ The quantization information for model inputs/outputs.
+ """
+
+ temp_dir = utils.tempdir()
+ extract_path = temp_dir.relpath("extract")
+ os.mkdir(extract_path)
+ with tarfile.TarFile(model_library_format_path) as f:
+ f.extractall(extract_path)
+
+ # Extract informations from the Model Library Format
+ graph_file = os.path.join(extract_path, "executor-config", "graph", "graph.json")
+ with open(graph_file, "r") as f:
+ # returns JSON object as a dictionary
+ graph_dict = json.load(f)
+
+ params_dict = {}
+ param_file = os.path.join(extract_path, "parameters", "default.params")
+ with open(param_file, "rb") as f:
+ params = tvm.runtime.load_param_dict(f.read())
+
+ # Map -> Python Dict
+ tmp_dict = {}
+ for (k, v) in params.items():
+ tmp_dict[k] = v
+
+ # Sort params for debugging
+ for k in sorted(tmp_dict.keys()):
+ params_dict[k] = tmp_dict[k]
+
+ src_dir = os.path.join(extract_path, "codegen", "host", "src")
+ # List of strings from Model Library Format C files
+ src_files = []
+ for filename in os.listdir(src_dir):
+ with open(os.path.join(src_dir, filename), "r") as fin:
+ src = fin.read()
+ src_files.append(src)
+
+ self.graph_ = graph_dict
Review comment:
think there are a couple more style fixes (e.g. self._graph and self._lib)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r616780971
##########
File path: apps/stm32/.clang-format
##########
@@ -0,0 +1,2 @@
+DisableFormat: true
Review comment:
just curious why you needed to do this?
##########
File path: apps/stm32/Makefile
##########
@@ -0,0 +1,90 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Top level Makefile for the STM32 microcontroller project.
+
+#
+# Set the path to your ARM Toolset.
+# Only gcc is supported at this time.
+#
+#export ARM_PATH ?=
+ifndef ARM_PATH
+$(error ARM_PATH must be set and point at your GCC ARM compiler installation)
+endif
+
+#
+# Set the path to your TVM compiler installation
+#
+#export TVM_PATH ?=
+ifndef TVM_PATH
+$(error TVM_PATH must be set and point at your TVM stack installation)
+endif
+
+#
+# Set the path to your CubeMX installation
+#
+#export X_CUBE_PATH ?=
+ifndef X_CUBE_PATH
+$(error X_CUBE_PATH must be set and point at your STM32 Cube Firmware installation)
+endif
+
+#export X_CUBE_TOOL_PATH ?=
+ifndef X_CUBE_TOOL_PATH
+$(error X_CUBE_TOOL_PATH must be set and point at your STM32 Cube Programmer installation)
+endif
+
+#
+# Setup path to your network generated files
+#
+#export MODEL_PATH =
+ifndef MODEL_PATH
+$(error MODEL_PATH must be set and point at your model implementation generated with the TVM compiler)
+endif
+
+BOARD ?= h747idisco
Review comment:
could you comment what's special about this board that makes it the default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-821086362
@areusch
How are you, Andrew
Can you help me out with this: we have some .h files for inclusion with the C code, they are checked via cpplint and the build fails with things like 'replace C-style casts ...'. How do you make these pass the lint checks ?
Thanks and cheers
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-853790415
@areusch @tqchen
Hello,
Addressed most of your review points, except of the last one below:
> on the question of 'crt_backend_api.c' -- i would like to avoid having to checkin a file such as 'src/runtime/crt/contrib/stm32/runtime.c' and instead rely on crt_backend_api.c to provide a common implementation. do you want to tackle that in this PR or in a follow-on?
If you look closer, the 'src/runtime/crt/contrib/stm32/runtime.c' implements the platform-specific functions, eg. the 'TVMPlatformMemoryAllocate'. I have kept an alternative implementation of these:
- TVMBackendAllocWorkspace (not used, under ifdef)
- TVMBackendFreeWorkspace (not used, under ifdef)
- TVMAPISetLastError (cannot use from c_runtime_api.c until next PR implemented)
- TVMGetLastError (cannot use from c_runtime_api.c until next PR implemented)
I will remove them at outcome of the next PR that will clean the c_backend_api.c. For now, prefer to keep these functions here.
> my comments here are around how the Python emitter is written--it would be great to ensure the style is in line with the rest of the codebase since it is going into 'python/tvm'.
Aligned the code along with your comments. The only thing I'd like to keep are my headers of type:
```
# ==========================================================
# get_input_tensor_name
# ==========================================================
```
I have moved any doc description from them into the python doc strings. However, I have a couple of scripts that rely on finding these headers. If removing them is not an absolute MUST, perhaps I can keep it. Let me know.
> we need to avoid repo bloat by not adding large test data (this model is MBs). can you download this from somewhere? see download_testdata in python/tvm/contrib/download.py.
Fixed that.
> thinking again about the SDK - TVM split, I see a bunch of stuff here that I think makes sense in TVM, e.g. 'struct ai_network_report'_ and dependencies. then I see some other stuff e.g. this ICC-specific logic here. It seems pretty hard to test this--wondering if it's necessary to place all of this in 'ai_platform.h'?
I need to verify and will get back on this point quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-874898522
Hello @areusch @tqchen
It seems that we are back to the idea of adding the STM32 codegen as a TVM app.
Like I have mentioned earlier in this PR, this way of integrating the STM32 is not entirely aligned with our vision of the STM32 target inside the TVM. We need to understand the implications/constraints of including the STM32 as a TVM app versus/along with it being distributed as a standalone tools package.
We will need to hold it here and re-discuss this PR status internally first.
Since we are in a European vacation period, it may take a while to get all necessary people's involvement.
I will let you know.
cheers
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-864529226
@areusch
Please review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-823497027
@stoa I think it depends how you plan to distribute support for targeting STM32 processors. do you plan to just include this in the STM X-Cube AI downloadable package, or is the intent to allow using STM32 processors with TVM directly, without the need to download the package from your site? if the latter, it'd help me to understand a bit more about your goal. I think it'd be preferable to place board-specific stuff and support utilities outside the TVM repo if possible (the TVM CI as you've seen is a bit lengthy and it'd be best to avoid unnecessarily including it in your development process).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-820719962
@stoa please ping me when you want me to take another look here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r626494235
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
How are you, Andrew @areusch
I have reorganized the code like so:
1. Removed the `apps/stm32`
2. Removed linker script code from the `emitter.py`
3. Removed the copy of `graph_runtime` from `src/runtime/stm32`
3. Moved the `src/runtime/stm32` to `src/runtime/crt/contrib/stm32`. I have chosen the `contrib` because it looks like other files in the directory are sort of common implementation, while the `stm32` is code-generator specific. If you prefer, not to use `contrib` for this - I will align the runtime and tests.
4. Moved the C part of our CI test to `tests/crt/contrib/stm32`
Now, I am looking at this comment:
> it seems like you could use the TVM C runtime's crt_backend_api.c and implement TVMPlatformMemoryAllocate(). Curious if that would be an option for you? btw, it would be totally fine with me if you wanted to split this file into a separate CRT library e.g. src/runtime/crt/backend to avoid including the rest of src/runtime/crt/common.
I have following observations:
1. Current TVM generates C code with these calls:
```
TVMAPISetLastError()
TVMGetLastError()
TVMBackendAllocWorkspace()
TVMBackendFreeWorkspace()
```
These would be good candidates for including with the `crt_backend_api.c`. Therefore, moving the `TVMAPISetLastError()` and the `TVMGetLastError()` there from the `crt_runtime_api.c`, and moving the `TVMBackendParallelLaunch()`, `TVMBackendRegisterSystemLibSymbol()` out to `crt_runtime_api.c`.
This way, the `crt_backend_api.c` would carry a minimal support required by the C target code generator.
2. Currently, the `TVMBackendAllocWorkspace` does some error handling. I would propose to leave the error handling to the caller, while the `TVMBackendAllocWorkspace` would just return `NULL` in case of error.
The reason is that the error handling and logging are platform-specific and even application-specific - eg. `malloc` does not do error handling, for example.
Handling this in the `TVMBackendAllocWorkspace` requires implementing the `TVMLogf` and the `TVMPlatformAbort`, which we will not be able to test. For example, in our `stm32` TVM test, I would have to implement the `x86` specific error handling, while for the Disco board I would set it to NOOP since the applications handles errors in its own way.
3. If we were to reduce the `crt_backend_api.c` to its minimal form as I have described above, perhaps it makes sense to have the application implement these few methods directly, i.e. no need to include the `tvm/crt/crt_backend_api.c` implementation, only the `crt_backend_api.h` header as interface ?
What do you think ?
Do not review the `tests/crt/contrib/stm32` yet - I will align it and clean after I have your opinion on the above.
cheers
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-856192323
@stoa sounds good to me!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r716183354
##########
File path: tests/micro/stm32/test_code_emitter.py
##########
@@ -27,7 +28,8 @@
# 1 = INFO messages are not printed
# 2 = INFO and WARNING messages are not printed
# 3 = INFO, WARNING, and ERROR messages are not printed
-os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
+if "TF_CPP_MIN_LOG_LEVEL" not in os.environ:
Review comment:
I am removing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-821334185
hi @stoa, try adding `// LINT__C_FILE` as a comment near the top of those files and that should fix that. sorry this isn't clear!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-824143000
@areusch
I have moved all ai_runner code from tvm/contrib/stm32 to the project,application directory apps/stm32. Only emitter.py remains in the contrib/stm32 directory.
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-821895633
@stoa try with: `docker/bash.sh ci_lint`. we should document this--it pulls the exact version from `Jenkinsfile`. When you see "Unable to find image abc locally," it typically means docker is downloading the image. These may be several GB (I think lint is ~2gb), so have some patience when this message appears. We don't update ci-lint very often, so it should just be a one-time thing, ish.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-856190683
@areusch
Hello, Andrew
Very good - our understanding is aligned. I had imagined the same schema as yours but currently the 'c_backend_api' includes the 'c_runtime_api' and I tried to keep the relation.
This simplifies the problem: all we need to do is to move the defines 'TVM_DLL' from the 'c_runtime_api' to the 'c_backend_api'; move the 'g_last_error' from the 'c_runtime_api' to the 'c_backend_api'; include the 'c_backend_api.h' from the 'c_runtime_api'.
Concerning the 'AI_API_' - sorry for the confusion, I do not pretend to push this prefix. The stm32 AI_API_ has a few additional defines on top of the 'c_runtime_api' TVM_DLL and TVM_WEAK. I was proposing to add those as TVM_ defines as well We can clean this up later. For now I will keep the stm32 defines with the ai_platform.h.
We prefer keep the source code with 'src/runtime/contrib/stm32' and I will reformat the code.
I will align the PR and we'll discuss the next step.
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r713990676
##########
File path: tests/crt/contrib/stm32/src/main.c
##########
@@ -0,0 +1,461 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include <inttypes.h>
+#include <math.h>
+#include <stdarg.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "ai_runtime_api.h"
+#include "network.h"
+#include "network_data.h"
+
+//
+// Network that we are testing
+//
+extern ai_model_info network_network;
+
+//
+// Dummy: for the runtime
+//
+uint32_t __models_section_start__ = (uint32_t)&network_network;
+uint32_t __models_section_end__ = (uint32_t)&network_network + sizeof(ai_model_info);
+
+static ai_model_info* _model_p = &network_network;
+
+//
+// Global handle to reference the instantiated NN
+//
+static ai_handle _network = AI_HANDLE_NULL;
+
+static uint8_t LoadInputImg(const char* filename, ai_tensor* input);
+static int32_t quantize_val(float val, ai_quantization_info* quant);
+static float dequantize_val(int32_t val, ai_quantization_info* quant);
+
+// =================================================================
+// Convert_Fixed_To_Float
+// =================================================================
+static float Convert_Fixed_To_Float(uint8_t data, int8_t fl) {
+ uint8_t val = data;
+ float x;
+ if (fl >= 0) {
+ x = ((float)val) / (float)(1 << fl); // NOLINT
+ } else {
+ x = ((float)val) / (float)(1 >> fl); // NOLINT
Review comment:
Thanks: copy-paste from a wrong place ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r720874174
##########
File path: python/tvm/micro/contrib/stm32/emitter.py
##########
@@ -0,0 +1,1369 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# pylint: disable=line-too-long
+
+"""Code emission for the STM32 targets."""
+
+import json
+import os
+import re
+import shutil
+import tarfile
+import textwrap
+
+from datetime import datetime
+
+import numpy as np
+
+import tvm
+from tvm.contrib import utils
+
+AI_API_VERSION_MAJOR = 1
+AI_API_VERSION_MINOR = 0
+AI_API_VERSION_MICRO = 0
+
+AI_TOOLS_REVISION = "v1"
+
+DBAR = "=" * 60
+
+
+def _fix_name(node_name):
+ """ Replace ':' with '_' in names like 'InputImg:0' """
+ return node_name.replace(":", "_")
+
+
+def get_input_tensor_name(node_name):
+ return _fix_name(node_name)
+
+
+def get_output_tensor_name(node_name, idx):
+ return _fix_name(node_name) + "_" + str(idx)
+
+
+def _get_node_args_name(node_name):
+ return _fix_name(node_name) + "_args"
+
+
+def _get_node_arg_types_name(node_name):
+ return _fix_name(node_name) + "_arg_type_ids"
+
+
+def _get_type_size(dltype):
+ if dltype in ("uint64", "int64"):
+ return 8
+ if dltype in ("uint32", "int32", "float32"):
+ return 4
+ if dltype in ("uint16", "int16"):
+ return 2
+ if dltype in ("uint8", "int8"):
+ return 1
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+C_TYPE_TO_DLTYPE = {
+ "uint64": "kDLUInt, 64, 1",
+ "int64": "kDLInt, 64, 1",
+ "float32": "kDLFloat, 32, 1",
+ "uint32": "kDLUInt, 32, 1",
+ "int32": "kDLInt, 32, 1",
+ "uint16": "kDLUInt, 16, 1",
+ "int16": "kDLInt, 16, 1",
+ "uint8": "kDLUInt, 8, 1",
+ "int8": "kDLInt, 8, 1",
+}
+
+
+def _get_type_data(dltype):
+ try:
+ return C_TYPE_TO_DLTYPE[dltype]
+ except KeyError:
+ raise ValueError(f"Data type {dltype} is not supported")
+
+
+def _get_aligned_offset(offset, dltype):
+ align = _get_type_size(dltype)
+ if offset % align != 0:
+ offset = offset + (align - offset % align)
+ return offset
+
+
+def _get_num_tensor_elts(shape):
+ size = 1
+ for dim in shape:
+ size = size * dim
+ return size
+
+
+def _get_tensor_size_bytes(dims, dltype):
+ size = _get_num_tensor_elts(dims)
+ return size * _get_type_size(dltype)
+
+
+def _preprocess_code(src):
Review comment:
Yes. Because math.h is missing (perhaps no longer missing in the newwest TVM), and because we have a CMSIS option that I will contribute later that needs more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-856072634
@stoa thanks for your detailed reply. i've tried to provide answers to all of your questions--let me know if a higher-bandwidth discussion would be helpful to align here (we can use the discord and document here)
> Do we agree on the following "software stack" ?:
i'd draw the software stack like so:
```
deployed inference_application generic app for host-driven inference/autotvm
| |
firmware-facing API |
| |
AOT [c] graph executor ----- c_runtime_api -- utvm_rpc_server
| | |
compiled operators [c,so] |
| |
c_backend_api.[h,c] <--------------+
|
platform-specific[platform.h, platform-specific implementation location]
```
Does this make sense? The c_backend_api is mainly a glue layer between the API exposed to operator functions and the platform-specific API, but it allows them to change independently of one another.
one note I'll make: right now "deployed inference application" as demonstrated/tested against Zephyr would use graph_executor and c_runtime_api. This is just due to the fact that we haven't finished developing the firmware-facing AOT interface yet.
> Do we agree that, in the AOT path, we do not want to include/compile the c_runtime_api.[h,c] ?
yeah definitely. i hope the above diagram clarifies my position on that, and apologies for any confusion. Stuff is changing fast right now.
> Do we agree that the 'g_last_error' (TVMGetSetLastError()) is set by the running inference application and not by the lower runtime layers: c_runtime_api|platform_specific ?
i think right now anything above c_backend_api could set that. c_runtime_api should be allowed to consume c_backend_api imo.
> I can propose to create some sort of 'c_runtime_defs.h' that would include platform-specific defines for:
>
>AI_API_ENTRY = TVM_DLL
>AI_API_ALIGNED
>AI_API_WEAK = TVM_WEAK
>etc.
>The platform-specific implementation can include this file then instead of the 'c_runtime_api.h'.
>Still the question of testing all different defines remain - but this is already the case with the 'c_runtime_api.h'.
i think one question is whether we want to promote the `AI_API_` prefix to be the TVM standard prefix. if so, this makes sense to me. however, i think such a proposal would need to be explicitly in an RFC on the discuss. I think we'd like to ultimately wind up with something compatible either way, but perhaps it makes sense to confine this `c_runtime_defs.h` to the STM32 code for now and promote it once the API settles. One thing is that since AI_ prefix is sort of used by STM32 code, using it here would mean that we may need to be careful about stepping on you--whereas keeping with `tvm_` prefix should be more safe.
> Moving to a different code location is not an issue, please let us know which directory is most suitable.
> And I also see that there is no reason for us to not reformat the C code - OK.
OK--here's what i think:
1. if it stays in `src/runtime/contrib/stm32`, it needs to get formatted to the TVM guidelines
2. if it lives outside of `src`, particularly in e.g. `apps/microtvm/stm32`, i don't think the formatting needs to follow TVM guidelines.
i've been working on the project-generator more this past week and will post it this week. in that branch, i've moved all of the zephyr code into [`apps/microtvm/zephyr/demo_runtime`](https://github.com/areusch/incubator-tvm/tree/project-generator/apps/microtvm/zephyr/demo_runtime)--in particular, the "emitter" is in `microtvm_api_server.py`. i'd ideally like to keep all the platform-specific stuff underneath `apps/microtvm/<platform>`, which allows TVM to interface a single target (the Model Library Format + microtvm API server) but continue to target multiple platforms. in exchange, code in that directory doesn't need to necessarily follow all of TVM's C formatting standards (e.g. there is iOS code in some of those directories).
i realize we had discussed this approach for stm32 before, but things weren't quite ready; plus you guys wanted to provide an improved API. we've discussed moving in the direction of improving microTVM's API such that the STM32 API can be provided by wrapping the microTVM API. when we reach that point, i'd like to move the STM32 code underneath `apps/microtvm/stm32` and use the project API (but, we may need to add to the API to allow generating libraries rather than whole projects, since the project generation for STM32 is handled by X-Cube tooling).
if you want to locate your code in `apps/microtvm/stm32` now and avoid reformatting, even if it doesn't use project generator, that would also be fine with me--let me know if this makes sense or not to you guys.
> Yes, intentional. We do not have a possibility to host these test-cases for downloading and they cannot be dowloaded from elsewhere.
If you want to push a PR to https://github.com/tlc-pack/web-data (see `testdata/microTVM`), we could host them there.
> Resuming: If we agree on re-organizing the 'c_runtime-api' as outlined above, I will align this PR and we will not need a separate PR in this case.
yeah that sounds fine to me. ideally it'd be a separate PR just to keep things neat, but it's not a big change so if it keeps things operationally simpler for you, feel free to include it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-846941749
Hello, Andrew @areusch
I have based the STM32 test build on the crt_backend_api.c using the TVMBackendAlloc/Free from there.
I am not moving the Set/GetLastError yet - this needs a separate RFC/PR because quite a few files are affected, and as you've pointed out we should take the opportunity to introduce the crt_error to the interface.
Please review this PR so we can terminate it and merge.
Meanwhile, I will take a closer look at the crt_error issue and the new RFC. I'll let you know quickly.
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on a change in pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on a change in pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#discussion_r631370429
##########
File path: src/runtime/stm32/graph_runtime.cc
##########
@@ -0,0 +1,529 @@
+/*
Review comment:
hi Arthur @stoa,
Ah I think I understand what's going on here...sorry for all of this confusion. You are stumbling into some problems we have not yet addressed for production inference use cases. There are a couple of things at play here:
### Error-handling paradigm and location of `TVMAPISetLastError()`.
`TVMAPISetLastError` and `TVMGetLastError` should be in `crt_backend_api.h` as you pointed out. However, microTVM has meanwhile adopted `tvm_crt_error_t` for many of its non-c_backend_api (and indeed, also in c_backend_api via calls to TVMPlatform*) functions.
> Returning the error codes from the crt_backend_api seems to be our commonly preferred way of handling errors here - I am very interested in this being proposed for the mTVM.
I concur with this opinion and this is what I'm going for with `tvm_crt_error_t`. I hadn't realized the misplacement of `TVMAPISetLastError`, since (from my reading) it's actually only used in generated code to handle assert fails, and I don't think we often use those in microTVM. Given this, I think it would be worth clarifying error handling in microTVM with a separate RFC. I don't think we should tackle that in this PR--perhaps a separate PR could:
1. Move `TVMAPISetLastError` and `TVMGetLastError` to `c_backend_api.h`.
2. Propose use of `tvm_crt_error_t` as the return value from TVM API functions, at least in the C runtime API. We could take this a step further and replace the `int` return value in the C++ runtime, but it may get complicated and `tvm_crt_error_t` is indeed compatible with the zero-indicates-success paradigm currently mandated by `c_{backend,runtime}_api.h`
3. Allocate a `tvm_crt_error_t` code to indicate assert failure, and direct users to `TVMGetLastError` to retrieve the error text.
I am happy to take on this RFC/PR if you would be okay waiting on it; or if you want to do it, I'm completely fine with that. Let me know if this direction makes sense to you.
### Handling `TVMBackendAllocWorkspace` errors
> I am not sure I understand this point. If the memory allocation fails, we have 2 choices:
>
> - The operator code catches it and SetLastError(), eventually with a detailed error code. I do not see why this is difficult or not clean - but perhaps I do not have enough vision here. In any case, this seems to have already been implemented with the TVM operators, Is this going to change ? Which RFC ?
> - The crt layer catches the error and handles via further hooks into the application code. We prefer to minimize such hooks.
> I am not sure why you have mentioned the CHECK_EQ macro ?
Here I was examining the existing implementation of `TVMBackendAllocWorkspace` in `src/runtime/crt/common/crt_backend_api.c`. That function is currently required to return NULL on error, so we would need to either:
1. change it to return `int` or `tvm_crt_error_t` to indicate errors
2. Make it follow the same paradigm as assert fails above
I prefer 1 here, since an error code can programmatically indicate what's wrong and is easier to log. Perhaps it's best to propose this as part of the RFC mentioned above.
I mentioned `CHECK_EQ` because that is the existing error-handling strategy. I think this calls `TVMPlatformAbort`, so I'm not sure it's the right one for inference deployment.
### Implementation of `TVMLogf` and `TVMPlatformAbort`
I agree at least `TVMLogf` should be optional in deployment. Likely, so should `TVMPlatformAbort`, though I think it remains to be seen how we can handle errors in higher-level libraries. I don't see operator code as needing to call `TVMPlatformAbort`, unless accelerator or other libraries need such a facility.
In any case, I agree it's fine to implement these as dummy for your production inference STM32 applications--once we resolve the `TVMBackendAllocWorkspace` error-handling strategy question above.
cc @tqchen regarding the API problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-925607823
@areusch
Hello, Andrew
Applied changes that you've requested.
Sorry for the comments from May 25 - some went with the revert and some I have not seen because they were hidden with the github page.
The last CI fails strangely, probably has nothing to do with my changes, I will re-run but this should not block the review.
cheers
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-823040256
@areusch
Thanks for the help. It's all good now. Tests pass.
Can you take a look at integrating ?
Arthur
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] areusch commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
areusch commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-918218038
@stoa and i discussed at length and are comfortable moving forward with merging this. there will be two phases to the STM32 integration:
phase 1: merge the custom codegen and runtime implemented here
--- µTVM embedded C API reaches feature parity with STM32 API --
phase 2: re-implement the STM32 APIs as thin wrappers around µTVM embedded C API. as this simplifies the STM32 codegen, consider removing it or moving it into a Project API implementation.
- in general TVM prefers to have a singular runtime choice (we currently have two: C++ for linux-like systems and C for bare-metal). in this case we are okay merging STM32 support given that the TVM embedded C API is actively being developed to feature parity with the STM32 API, the STM32 runtime component is minimal, and having STM32 support in the codebase serves as a second point of reference in designing middle compiler functionality to support a robust embedded C API.
- target-wise: we can support dedicated stm32 targets through e.g. `tvm.target.micro(board="stm32f746zg_disco")`
- test-wise: we can unit test the code-generator and runtimes as needed and checkin a basic x86-based integration test. hardware-in-the-loop testing will be addressed as µTVM begins to consider how to tackle this as a community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@tvm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [tvm] stoa commented on pull request #7742: Contributing the STM32 port
Posted by GitBox <gi...@apache.org>.
stoa commented on pull request #7742:
URL: https://github.com/apache/tvm/pull/7742#issuecomment-853790415
@areusch @tqchen
Hello,
Addressed most of your review points, except of the last one below:
> on the question of 'crt_backend_api.c' -- i would like to avoid having to checkin a file such as 'src/runtime/crt/contrib/stm32/runtime.c' and instead rely on crt_backend_api.c to provide a common implementation. do you want to tackle that in this PR or in a follow-on?
If you look closer, the 'src/runtime/crt/contrib/stm32/runtime.c' implements the platform-specific functions, eg. the 'TVMPlatformMemoryAllocate'. I have kept an alternative implementation of these:
- TVMBackendAllocWorkspace (not used, under ifdef)
- TVMBackendFreeWorkspace (not used, under ifdef)
- TVMAPISetLastError (cannot use from c_runtime_api.c until next PR implemented)
- TVMGetLastError (cannot use from c_runtime_api.c until next PR implemented)
I will remove them at outcome of the next PR that will clean the c_backend_api.c. For now, prefer to keep these functions here.
> my comments here are around how the Python emitter is written--it would be great to ensure the style is in line with the rest of the codebase since it is going into 'python/tvm'.
Aligned the code along with your comments. The only thing I'd like to keep are my headers of type:
```
# ==========================================================
# get_input_tensor_name
# ==========================================================
```
I have moved any doc description from them into the python doc strings. However, I have a couple of scripts that rely on finding these headers. If removing them is not an absolute MUST, perhaps I can keep it. Let me know.
> we need to avoid repo bloat by not adding large test data (this model is MBs). can you download this from somewhere? see download_testdata in python/tvm/contrib/download.py.
Fixed that.
> thinking again about the SDK - TVM split, I see a bunch of stuff here that I think makes sense in TVM, e.g. 'struct ai_network_report'_ and dependencies. then I see some other stuff e.g. this ICC-specific logic here. It seems pretty hard to test this--wondering if it's necessary to place all of this in 'ai_platform.h'?
I need to verify and will get back on this point quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org