You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tvm.apache.org by GitBox <gi...@apache.org> on 2019/12/22 17:29:00 UTC
[GitHub] [incubator-tvm] FrozenGene opened a new pull request #4564: [Doc]
Introduction to module serialization
FrozenGene opened a new pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564
Follow up of https://github.com/apache/incubator-tvm/pull/4532, this doc for introducing module serialization and implementation details how we achieve it.
@tqchen
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-570999273
Thanks the comment and suggestion of @zhiics @yzhliu @yangjunpro I will address the comment in the following days.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on issue #4564: [Doc] Introduction
to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-582726010
LGTM now. I will wait till tomorrow to see if there are any more comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361863270
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
Review comment:
.dll for Windows or .so for Linux
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics merged pull request #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics merged pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on issue #4564: [Doc] Introduction
to module serialization
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-569098656
It would be great if we can add a figure explaining the storage relation and data structure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361863398
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
Review comment:
```suggestion
The following code is used to construct ``ModuleSerializer``:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r365336690
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Firstly, Let us build one ResNet-18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
Review comment:
```suggestion
4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene edited a comment on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene edited a comment on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-573016580
Have addressed the comments. I also add some words of `PackImportToC` existed in the PR #4657 , so we should be better handle it after that PR in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367162864
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
Review comment:
```suggestion
``binary_blob_size`` is the number of blobs we will have in this
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361857238
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
Review comment:
Firstly, when we load it back, we will do the follow logic: ```cpp for(int i= 0; i < binary_blob_size; ++i) { Read(key); do something for this key(like call module.loadbinary_tkey or read _import_tree_row_ptr / _import_tree_child_indices) }``` . So if we don’t have one slot for _import_tree (i.e. don’t keep the same blob layout as before) ,we can not read it back smoothly. So, you could imagine we have one module, whose type key is _import_tree, its serialization logic is to save _import_tree_row_ptr / _import_tree_child_indices. Secondly, _import_tree will indicate our library (deploy.so) is using new export mechanism and we could use _import_tree to reconstruct import relationship. If we don’t have this key, we will fallback to old behavior. This design could make us keep backward compatible. That is to say, in the new runtime system, your old exported library could work perfectly too. Does it make sense to you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862939
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
Review comment:
```suggestion
OpenCL or anything else. We don't restrict the module type here.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on issue #4564: [Doc] Introduction
to module serialization
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-569447404
sounds good, would love to see others' thoughts, @zhiics @yzhliu @ajtulloch @ZihengJiang
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862874
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
Review comment:
Should be "llvm modules and C modules", and C source module is actually not a DSO module, we need to invoke compilation later to link them together with the DSO module.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yzhliu commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yzhliu commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r363147303
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
Review comment:
If -> Once
not very clear to me how "creating dev.cc" and "compile into one dll" are related. mind elaborate a bit more?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-569212188
> > It would be great if we can add a figure explaining the storage relation and data structure.
>
> Will do.
@tqchen need to double check to make sure I understand correctly.
The doc describes:
- Module import relationship. i.e. LLVM imports cuda module. <- No need to have figure in my opinion as it only have two modules and is a simple relationship
- Serialization Format.
```cpp
binary_blob_size
binary_blob_type_key
binary_blob_logic
...
_import_tree
_import_tree_logic
```
<- No need to have figure, as this simple table could describe cleanly.
- Mark Module Index. We have LLVM Module (0, root module) <- CUDA Module (1). <- No need to have figure.
- import tree of CSR storage. Each row is parent index, each col is child indices. <- We need to have figure to describe what is CSR module index mapping status. Like this figure: https://img-blog.csdn.net/2018092717062821?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2Jvcmw=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene edited a comment on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene edited a comment on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-569046930
Anyone has interest in helping to review and give feedback? @zhiics @yzhliu @tqchen Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-573016580
Have addressed the comments. I also add some words of `PackImportToC` existed in the PR #4657 , so we should handle it after that PR in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
tqchen commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r360722448
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,206 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+**Author**: `Zhao Wu <https://github.com/FrozenGene>`_
Review comment:
Given that this is part of doc as opposed to tutorial, let us not include author's name for now(to be consistent with the rest part of the doc) as we want to encourage future collaborations and improvements. Of course the contribution is part of the git commit history :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on issue #4564: [Doc] Introduction
to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-583199805
Thanks everyone. This is now merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on issue #4564: [Doc] Introduction
to module serialization
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-568292243
Would be great if we can get more readers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r365336043
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Firstly, Let us build one ResNet-18 workload for GPU as our example firstly.
Review comment:
```suggestion
Let us build one ResNet-18 workload for GPU as an example first.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361864082
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could get using ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
+``SaveToBinary`` function to serialize blob into binary. So, for most
+modules, they should implement ``SaveToBinary`` function. But like DSO
+module, we don't implement ``SaveToBinary``, we also don't some special
Review comment:
We probably want to elaborate a bit about when users need to implement SaveToBinary. It is mainly implemented for device modules not for the host module such as DSO
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862728
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
Review comment:
```suggestion
First, let us build one ResNet-18 workload for GPU as our example.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361850092
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
Review comment:
the child indices "is" ==> the child "corresponds to"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-569168684
> It would be great if we can add a figure explaining the storage relation and data structure.
Will do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361850415
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
Review comment:
Maybe a diagram or figure could help elaborate this section more clearly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361850286
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
Review comment:
It looks that "_import_tree" is a meta data, so just a little bit curious why it also occupies another slot as the imported modules(LLVM/CUDA).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367747814
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could be got by ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
Review comment:
Yes. I will rephrase it next.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862805
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
Review comment:
remove extra blank line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene edited a comment on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene edited a comment on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-569212188
> > It would be great if we can add a figure explaining the storage relation and data structure.
>
> Will do.
@tqchen need to double check to make sure I understand correctly.
The doc describes:
- Module import relationship. i.e. LLVM imports cuda module. <- No need to have figure in my opinion as it only have two modules and is a simple relationship
- Serialization Format.
```cpp
binary_blob_size
binary_blob_type_key
binary_blob_logic
...
_import_tree
_import_tree_logic
```
<- No need to have figure, as this simple table could describe cleanly.
- Mark Module Index. We have LLVM Module (0, root module) <- CUDA Module (1). <- No need to have figure.
- import tree of CSR storage. Each row is parent index, each col is child indices. <- We need to have figure to describe what is CSR module index mapping status. Like this figure: https://img-blog.csdn.net/2018092717062821?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2dhb2Jvcmw=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r365336370
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Firstly, Let us build one ResNet-18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named as ``dev.cc``
Review comment:
```suggestion
Once we have imported modules, we will create one file named ``dev.cc``
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r363155024
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
Review comment:
The better description maybe should be (so that we could embed the binary blob data of import modules into one dll). For example, we have one CUDA module, which has one function named as SaveToBinary to serialize related data of CUDA module (like function information table, binary data and so on) into binary. Then we will translate the binary data into C program `const char __tvm_dev_blob[] = {0x...0x...0x...}`. To embed this binary information, we will write the C program into dev.cc, which will be compiled into shared library by compiler (using -fPIC -shared). Finally we embed the CUDA module binary information in the shared library’s global binary section, the symbol name is __tvm_dev_blob.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361863477
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
Review comment:
```suggestion
using DFS and create index for them. Note the root
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361863371
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
Review comment:
```suggestion
For better understanding, let us dig the implementation of this class a little deeper.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen commented on issue #4564: [Doc] Introduction
to module serialization
Posted by GitBox <gi...@apache.org>.
tqchen commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-582719710
ping @zhiics @yzhliu @FrozenGene @yangjunpro please followup on this thread :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361849686
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
Review comment:
no matter "whether" it is CPU or GPU
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862815
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
Review comment:
remove extra blank line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r360749319
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,206 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+**Author**: `Zhao Wu <https://github.com/FrozenGene>`_
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361863550
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
Review comment:
correspond
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yzhliu commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yzhliu commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r363146820
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
Review comment:
```suggestion
2. Once we have DSO modules, we will call ``save`` function to save them into files.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367205537
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could be got by ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
Review comment:
Not correct completely. My meaning is we will do logic handling for different binary blob type (like CUDA, LLVM, import tree ...). Different blob will have different logic. Like CUDA, in the ``binary_blob_logic``, we will do serialization of the binary blob using ``SaveToBinary``. However, for LLVM, we just to write `_lib` to indicate this is a DSO module. For `_import_tree`, we will write `_import_tree` and one CSR data structure recoding module import relationship. So I write will do the ``binary_blob_logic`` and following sentences to say, most of time we will call ``SaveToBinary`` to do serialization of this binary blob, but it is not all cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r365337738
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Firstly, Let us build one ResNet-18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They will call one same function ``SerializeModule`` to do module serialization. In ``SerializeModule``
Review comment:
```suggestion
They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361863484
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
Review comment:
```suggestion
module is fixed at location 0. In our example, we have module relationship:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862708
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
Review comment:
Align *
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367550915
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could be got by ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
Review comment:
oaky, so can rephrase it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862798
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
Review comment:
Remove extra blank line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361863340
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
Review comment:
```suggestion
It will take ``module`` to do some initialization work, like marking module index.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yzhliu commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yzhliu commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r363148082
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could get using ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
+``SaveToBinary`` function to serialize blob into binary. So, for most
+modules, they should implement ``SaveToBinary`` function. But like DSO
+module, we don't implement ``SaveToBinary``, we also don't some special
+handling, so the ``binary_blob_logic`` for DSO module currently
+is empty in fact.
+
+Finally, we will write one key ``_import_tree``. Unless our module only
+have one DSO module and it is in the root, otherwise we will always
+have this key. It is used for reconstruct the module import relationship
+when we load the exported library back as said before. The
+``import_tree_logic`` is just write ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` into stream.
+
+After this step, we will pack it into a symbol
+``runtime::symbol::tvm_dev_mblob`` that can be recovered in the dynamic
+libary.
+
+Now, we complete the serialization part. As you have seen, we could
+support arbitrary modules to import ideally.
+
+****************
+Deserialization
+****************
+
+The entrance API is ``tvm.module.load``. This function
+is to call ``_LoadFromFile`` in fact. If we dig it a little deeper, this is
+``Module::LoadFromFile``. In our example, the file ``deploy.so``,
+according to the function logic, we will call ``module.loadfile_so`` in
+``dso_library.cc``. The key is here:
+
+.. code:: c++
+
+ // Load the imported modules
+ const char* dev_mblob = reinterpret_cast<const char*>(lib->GetSymbol(runtime::symbol::tvm_dev_mblob));
+ Module root_mod;
+ if (dev_mblob != nullptr) {
+ root_mod = ProcessModuleBlob(dev_mblob, lib);
+ } else {
+ // Only have one single DSO Module
+ root_mod = Module(n);
+ }
+
+As said before, we will pack the blob into the symbol
+``runtime::symbol::tvm_dev_mblob``. During deserialization part, we will
+inspect it. If we have it, we will call ``ProcessModuleBlob``, whose logic like this:
Review comment:
do you mean "if we have the properly serialized blob" ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367163440
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
Review comment:
There will be three blobs in our example which are created for LLVM module, CUDA module, and ``_import_tree``, respectively.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367164474
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could be got by ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
Review comment:
I don't quite get this. Do you mean "Next, we work on the serialization of the binary blob"?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367163646
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
Review comment:
This sentence seems broken as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361862922
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
Review comment:
```suggestion
3. Next, we will check whether we have imported modules, such as CUDA,
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] tqchen edited a comment on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
tqchen edited a comment on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-568292243
Would be great if we can get more readers to help review
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on issue #4564: [Doc]
Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on issue #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#issuecomment-569046930
Anyone has interest to help to review and give feedback? @zhiics @yzhliu @tqchen Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361857238
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
Review comment:
Firstly, when we load it back, we will do the follow logic: ```cpp for(int i= 0; i < binary_blob_size; ++i) { Read(key); do something for this key(like call module.loadbinary_tkey or read _import_tree_row_ptr / _import_tree_child_indices) }``` . So if we don’t have one slot for _import_tree (i.e. don’t keep the same blob layout as before) ,we can not read it back smoothly like now. So, you could imagine we have one module, whose type key is _import_tree, its serialization logic is to save _import_tree_row_ptr / _import_tree_child_indices. Secondly, _import_tree will indicate our library (deploy.so) is using new export mechanism and we could use _import_tree to reconstruct import relationship. If we don’t have this key, we will fallback to old behavior. This design could make us keep backward compatible. That is to say, in the new runtime system, your old exported library could work perfectly too. Does it make sense to you?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361850386
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could get using ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
+``SaveToBinary`` function to serialize blob into binary. So, for most
+modules, they should implement ``SaveToBinary`` function. But like DSO
+module, we don't implement ``SaveToBinary``, we also don't some special
+handling, so the ``binary_blob_logic`` for DSO module currently
+is empty in fact.
+
+Finally, we will write one key ``_import_tree``. Unless our module only
+have one DSO module and it is in the root, otherwise we will always
+have this key. It is used for reconstruct the module import relationship
+when we load the exported library back as said before. The
+``import_tree_logic`` is just write ``import_tree_row_ptr_`` and
Review comment:
"is just write" ==>"just writes"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r363155120
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could get using ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
+``SaveToBinary`` function to serialize blob into binary. So, for most
+modules, they should implement ``SaveToBinary`` function. But like DSO
+module, we don't implement ``SaveToBinary``, we also don't some special
+handling, so the ``binary_blob_logic`` for DSO module currently
+is empty in fact.
+
+Finally, we will write one key ``_import_tree``. Unless our module only
+have one DSO module and it is in the root, otherwise we will always
+have this key. It is used for reconstruct the module import relationship
+when we load the exported library back as said before. The
+``import_tree_logic`` is just write ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` into stream.
+
+After this step, we will pack it into a symbol
+``runtime::symbol::tvm_dev_mblob`` that can be recovered in the dynamic
+libary.
+
+Now, we complete the serialization part. As you have seen, we could
+support arbitrary modules to import ideally.
+
+****************
+Deserialization
+****************
+
+The entrance API is ``tvm.module.load``. This function
+is to call ``_LoadFromFile`` in fact. If we dig it a little deeper, this is
+``Module::LoadFromFile``. In our example, the file ``deploy.so``,
+according to the function logic, we will call ``module.loadfile_so`` in
+``dso_library.cc``. The key is here:
+
+.. code:: c++
+
+ // Load the imported modules
+ const char* dev_mblob = reinterpret_cast<const char*>(lib->GetSymbol(runtime::symbol::tvm_dev_mblob));
+ Module root_mod;
+ if (dev_mblob != nullptr) {
+ root_mod = ProcessModuleBlob(dev_mblob, lib);
+ } else {
+ // Only have one single DSO Module
+ root_mod = Module(n);
+ }
+
+As said before, we will pack the blob into the symbol
+``runtime::symbol::tvm_dev_mblob``. During deserialization part, we will
+inspect it. If we have it, we will call ``ProcessModuleBlob``, whose logic like this:
Review comment:
I mean "if we have symbol of runtime::symbol::tvm_dev_mblob"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361850344
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could get using ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
+``SaveToBinary`` function to serialize blob into binary. So, for most
+modules, they should implement ``SaveToBinary`` function. But like DSO
+module, we don't implement ``SaveToBinary``, we also don't some special
+handling, so the ``binary_blob_logic`` for DSO module currently
+is empty in fact.
+
+Finally, we will write one key ``_import_tree``. Unless our module only
+have one DSO module and it is in the root, otherwise we will always
Review comment:
"have"==>"has"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r363155024
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
Review comment:
The better description maybe should be (so that we could embed the binary blob data of import modules into one dll). For example, CUDA module, we will have one function named as SaveToBinary, we will use it to serialize related data of CUDA module (like function information table, binary data and so on) into binary. Then we will translate the binary blob data in C program `const char __tvm_dev_blob[] = {0x...0x...0x...}`, finally we will write this into dev.cc, which will be compiled into shared library by compiler (using -fPIC -shared).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] FrozenGene commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
FrozenGene commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367205537
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could be got by ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
Review comment:
Not correct completely. My meaning is we will do logic handling for different binary blob type (like CUDA, LLVM, import tree ...). Different blob will have different logic. Like CUDA, in the ``binary_blob_logic``, we will do serialization of the binary blob using ``SaveToBinary``. However, for LLVM, we just to write `_lib` to indicate this is a DSO module. For `_import_tree`, we will write `_import_tree` and one CSR data structure recoding module import relationship. So I write we will do the ``binary_blob_logic`` and following sentences to say, most of time we will call ``SaveToBinary`` to do serialization of this binary blob, but it is not all cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] zhiics commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
zhiics commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r367165589
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,227 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter whether it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+*********************
+Module Export Example
+*********************
+
+Let us build one ResNet-18 workload for GPU as an example first.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+*************
+Serialization
+*************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM modules and C modules)
+
+2. Once we have DSO modules, we will call ``save`` function to save them into files.
+
+3. Next, we will check whether we have imported modules, such as CUDA,
+ OpenCL or anything else. We don't restrict the module type here.
+ Once we have imported modules, we will create one file named ``dev.cc``
+ (so that we could embed the binary blob data of import modules into one dynamic shared library),
+ then call function ``_PackImportsToLLVM`` or ``_PackImportsToC`` to do module serialization.
+
+4. Finally, we call ``fcompile`` which invokes ``_cc.create_shared`` to get
+ dynamic shared library.
+
+.. note::
+ 1. For C source modules, we will compile them and link them together with the DSO module.
+
+ 2. Use ``_PackImportsToLLVM`` or ``_PackImportsToC`` depends on whether we enable LLVM in TVM.
+ They achieve the same goal in fact.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in the ``_PackImportsToLLVM`` or ``_PackImportsToC``.
+They both call ``SerializeModule`` to serialize the runtime module. In ``SerializeModule``
+function, we firstly construct one helper class ``ModuleSerializer``. It will take ``module`` to do some
+initialization work, like marking module index. Then we could use its ``SerializeModule`` to serialize module.
+
+For better understanding, let us dig the implementation of this class a little deeper.
+
+The following code is used to construct ``ModuleSerializer``:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Note the root module is fixed at
+location 0. In our example, we have module relationship like this:
+
+.. code:: c++
+
+ llvm_mod:imported_modules
+ - cuda_mod
+
+So LLVM module will have index 0, CUDA module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used to restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices correspond to its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could be got by ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
+``SaveToBinary`` function to serialize blob into binary.
+
+.. note::
+ When to implement ``SaveToBinary``, it depends on circumstances.
Review comment:
Whether or not it is required to implement the ``SaveToBinary`` virtual function depends on how the module is used. For example, there are some typical cases as following.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361850360
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could get using ``module->type_key()``.
+
+Next we will do the ``binary_blob_logic``. Normally, we will call
+``SaveToBinary`` function to serialize blob into binary. So, for most
+modules, they should implement ``SaveToBinary`` function. But like DSO
+module, we don't implement ``SaveToBinary``, we also don't some special
+handling, so the ``binary_blob_logic`` for DSO module currently
+is empty in fact.
+
+Finally, we will write one key ``_import_tree``. Unless our module only
+have one DSO module and it is in the root, otherwise we will always
+have this key. It is used for reconstruct the module import relationship
Review comment:
"for reconstruct"==>"to reconstruct"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yangjunpro commented on a change in pull request
#4564: [Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yangjunpro commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r361850053
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
Review comment:
"for restore" ==> "to restore"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-tvm] yzhliu commented on a change in pull request #4564:
[Doc] Introduction to module serialization
Posted by GitBox <gi...@apache.org>.
yzhliu commented on a change in pull request #4564: [Doc] Introduction to module serialization
URL: https://github.com/apache/incubator-tvm/pull/4564#discussion_r363148849
##########
File path: docs/dev/introduction_to_module_serialization.rst
##########
@@ -0,0 +1,211 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+Introduction to Module Serialization
+====================================
+
+When to deploy TVM runtime module, no matter it is CPU or GPU, TVM only needs one single DLL.
+The key is our unified module serialization mechanism. This document will introduce TVM module
+serialization format standard and implementation details.
+
+***************************************
+Module Export Example
+***************************************
+
+Let us build one resnet18 workload for GPU as our example firstly.
+
+.. code:: python
+
+ from tvm import relay
+ from tvm.relay import testing
+ from tvm.contrib import util
+ import tvm
+
+ # Resnet18 workload
+ resnet18_mod, resnet18_params = relay.testing.resnet.get_workload(num_layers=18)
+
+ # build
+ with relay.build_config(opt_level=3):
+ _, resnet18_lib, _ = relay.build_module.build(resnet18_mod, "cuda", params=resnet18_params)
+
+ # create one tempory directory
+ temp = util.tempdir()
+
+ # path lib
+ file_name = "deploy.so"
+ path_lib = temp.relpath(file_name)
+
+ # export library
+ resnet18_lib.export_library(path_lib)
+
+ # load it back
+ loaded_lib = tvm.module.load(path_lib)
+ assert loaded_lib.type_key == "library"
+ assert loaded_lib.imported_modules[0].type_key == "cuda"
+
+
+**************
+Serialization
+**************
+
+The entrance API is ``export_library`` of ``tvm.module.Module``.
+Inside this function, we will do the following steps:
+
+1. Collect all DSO modules (LLVM module or C module)
+
+
+2. If we have DSO modules, we will call ``save`` function to save them into files.
+
+
+3. Next, we will check whether we have imported modules. Like CUDA,
+ OpenCL or anything else, we don't restrict the module type here.
+ If we have imported modules, we will create one file named as ``dev.cc``
+ (so that we could compile into one dll), then call one function
+ ``_PackImportsToC`` to do module serialization.
+
+
+4. Finally, we use ``fcompile`` to call ``_cc.create_shared`` to get
+ dll.
+
+***************************************************
+Under the Hood of Serialization and Format Standard
+***************************************************
+
+As said before, we will do the serialization work in ``_PackImportsToC``.
+Inside this function, we firstly construct one helper class ``ModuleSerializer``.
+It will take ``module`` to do some initialize work, like marking module index.
+Then we could use its ``SerializeModule`` to serialize module.
+
+For the sake of understanding better, let us dig the implementation of this class a little deeper.
+
+When we construct ``ModuleSerializer``, we could see this code:
+
+.. code:: c++
+
+ explicit ModuleSerializer(runtime::Module mod) : mod_(mod) {
+ Init();
+ }
+ private:
+ void Init() {
+ CreateModuleIndex();
+ CreateImportTree();
+ }
+
+In ``CreateModuleIndex()``, We will inspect module import relationship
+using DFS and create index for them. Here, we have one invariance: root
+module is always at location 0. In our example, we have module relationship:
+``LLVM Module <- CUDA Module.`` So LLVM module will have index 0, CUDA
+module will have index 1.
+
+After constructing module index, we will try to construct import tree (``CreateImportTree()``),
+which will be used for restore module import relationship when we load
+the exported library back. In our design, we use CSR format to store
+import tree, each row is parent index, the child indices is its children
+index. In code, we use ``import_tree_row_ptr_`` and
+``import_tree_child_indices_`` to represent them.
+
+After initialization, we could serialize module using ``SerializeModule`` function.
+In its function logic, we will assume the serialization format like this:
+
+.. code:: c++
+
+ binary_blob_size
+ binary_blob_type_key
+ binary_blob_logic
+ ...
+ _import_tree
+ _import_tree_logic
+
+``binary_blob_size`` is how many blobs we will have in this
+serialization step. In our example, the number will equal to 3. One for
+LLVM module, one for CUDA module, one for ``_import_tree``.
+
+Then we will write the ``binary_blob_type_key``, for LLVM module / C
+module, the blob type key is ``_lib``. For CUDA module, it is
+``cuda``, which could get using ``module->type_key()``.
Review comment:
```suggestion
``cuda``, which can be got by ``module->type_key()``.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services